Executive Summary—
Evaluation of NRC’s Industrial Research Assistance Program (IRAP)

NRC’s Policy, Planning and Assessment Directorate carried out an evaluation of IRAP in 2001-2002 in accordance with the Treasury Board Secretariat’s evaluation policy and with endorsement by NRC’s Governing Council.  The evaluation addressed the performance of IRAP’s Technological and Advisory Services and Non-Repayable Contributions to small and medium-sized enterprises (SMEs)  for Research & Development (R&D), as well as the performance of the Canadian Technology Network (CTN) for the period of 1996-1997 to 2000-2001, inclusive.  The evaluation did not assess IRAP’s repayable contribution program (Pre-commercialization Assistance)—this is being evaluated in the Fall of 2002 by Technology Partnership Canada in collaboration with IRAP.  

The evaluation focussed on the following issues:
•	Relevance:  the extent to which IRAP continues to be consistent with the federal government’s innovation priorities, and addresses the actual needs of SMEs; 
•	Success:  the extent to which IRAP has been successful in meeting its program objectives and intended reach; 
•	Effectiveness of Program Delivery:  the extent to which IRAP’s delivery structure is effective in achieving results; and 
•	Auditor General Follow-up:  IRAP’s role in Canada’s innovation system, the extent of the program’s incrementality and performance measurement practices.

Multiple lines of evidence were used to conduct the evaluation, including:
•	A document and data review on innovation, Canadian SMEs and IRAP clients;
•	In-person and telephone interviews with 120 key IRAP stakeholders;
•	A survey with 684 IRAP clients who received IRAP funding for projects during the period of 1996-2001 (extrapolated to 9,158 unique IRAP clients and 14,564 unique projects over 1996-2001 period);
•	A socio-economic analysis study of 26 IRAP client projects;
•	An international comparison study with six programs abroad;
•	An innovation impact analysis and a comparison of innovative characteristics (with Statistics Canada’s 1999 Innovation Survey); 
•	Surveys with 145 CTN clients (CTN Member Organizations and SMEs); and
•	A national and regional delivery analysis.
IRAP’s Context
IRAP has existed for close to 60 years to help Canadian SMEs improve their technological performance through innovation.  IRAP’s mandate and mission are ‘to stimulate wealth creation for Canada through technological innovation’ and ‘to stimulate innovation in Canadian SMEs’.  IRAP’s role vis-à-vis NRC’s Vision to 2006 is to achieve excellence and leadership in support of Canadian SMEs to innovate locally and compete globally.  Within the federal government’s innovation agenda, IRAP is positioned to address Canada’s knowledge performance challenge by helping to create the conditions for the production of new knowledge and/or its commercialization. 

IRAP’s annual budget is approximately $150M:
•	43% for Non-Repayable Contributions for R&D Activities;
•	20% for IRAP-Technology Partnership Canada (Pre-commercialization Assistance)
•	16% for Contributions to IRAP Network Members 
•	15% for operations and salaries;
•	3% for Canadian Technology Network; and
•	3% for Youth Initiatives.

IRAP’s Technological and Advisory Services and Non-Repayable Contributions for R&D Activities are delivered by 256 Industrial Technology Advisors (ITAs) located in 90 communities across Canada.  Approximately 70% of ITAs are employed in over 100 public and private sector organizations known as IRAP Network Members.  IRAP is also supported by operational staff--approximately 141 FTEs located at IRAP’s national and regional offices.
Key Evaluation Findings
The evaluation’s overall conclusion is that IRAP continues to be relevant to Canadian SMEs and to the federal government’s innovation priorities, and has been successful in meeting its mandate and mission.  

IRAP has helped to increase the innovation and financial performance of Canadian SMEs:
•	Approximately 12,364 IRAP funded projects culminated in 39,186 new/ significantly improved products/ services or processes over 1996-2001—approximately 3.2 innovations per IRAP funded project;
•	Approximately $11.3B actual sales revenues are linked to IRAP-assisted innovations--$4.2B attributable to IRAP over 1996-2001 and equivalent to 11 times IRAP’s total contributions to client projects during this period;
•	Approximately $37.6B forecasted future sales revenues are linked to IRAP-assisted innovations—$14B attributable to IRAP during the remainder of clients’ innovation life cycles; and
•	Approximately 32,600 actual additional jobs are linked to IRAP-assisted innovations—12,025 jobs attributable to IRAP over 1996-2001 and equivalent to $32,000 of IRAP contributions per job created during this period.

IRAP has also provided value and wealth creation for Canada:
•	Approximately 37% of IRAP client innovations are considered ‘World Firsts’, 66% ‘Firsts in Canada’, and 96% ‘Firsts in the Firm;
•	IRAP contributions to client projects are associated with total investments for all phases of the clients’ innovation projects equivalent to approximately 12.5 times IRAP’s contributions--firms’ own equity make up almost 50% of these investments;
•	Estimates from the evaluation’s socio-economic analysis study, indicate that IRAP provides the public with value for money--IRAP contributions provide a return to the federal government in the form of present and future corporate income taxes valued at approximately 11 times the value of IRAP’s contributions; and
•	IRAP is recognized by clients as the top government technology support program and the top external source of information (outside of firms’ supply chain and publicly available information). 
Strategic Considerations and Recommendations
The evaluation also concludes that some key strategic issues should be considered by NRC and IRAP Management to ensure the continued success and relevance of the program.  The following are the issues and related conclusions, recommendations and management responses:

1.0	Address Client Funding Needs:  The evaluation indicates that IRAP clients operate in very technical and competitive environments, consider project costs and financing as top barriers to innovation, and place a high level of importance on IRAP funding for research and pre-competitive development.  In addition, the evaluation indicates that clients who receive more funding tend to have higher success rates (e.g., additional jobs, innovations linked to IRAP assistance).  Within this context, there is a continued need for IRAP funding but also a need to increase the level of funding provided per client.  Although IRAP’s total budget has increased over time, IRAP’s total non-repayable contribution budget for R&D activities has not increased since 1998.  In addition, the total number of new IRAP projects funded per year has decreased (from 3,537 in 1996-97 to 2,339 in 2000-01) and the average amount of yearly funding per project has increased over time but not to the same level as in 1996-97 ($30,276 in 1996-97 compared to $27,897 in 2000-01).

1.1	Recommendation:  The evaluation recommends that IRAP increase the level of funding per client for research and development projects.  In support of this, consideration should be given to seeking an increase to IRAP’s non-repayable contribution budget in order to reach more clients and provide more funding per client.  Consideration should also be given to exploring “top up” programs to increase the level of funding available to clients.  Without an increase in IRAP’s contribution budget, IRAP will have to make trade-offs--the key one being to reduce its client reach (i.e., in order to provide more funding per client)--to maximize IRAP’s value to SMEs and Canada.

1.2	IRAP Response:  IRAP agrees with the need to increase its level of funding per client for research and development projects; in fact, IRAP has begun to put this strategy into effect and with the resulting trade-off of reaching fewer clients.  As part of its new Strategic Plan and with additional funding, IRAP is expecting to double its client reach over the next five years (from 12,000 to 24,000 clients by 2007).  NRC is requesting additional funding for IRAP from the Federal Government to ensure that the program can assist more SMEs increase their innovation capabilities over the next few years and is considering innovative approaches to increase the level of funding available per client.  IRAP is also working with other sources of funding such as venture capital, BDC and regional agencies to facilitate SME access to funding for innovation related activities.  

2.0	Increase Value of IRAP Advice and Cost-Effectiveness:  The evaluation indicates that clients perceive IRAP advice as substantially less important and less incremental to their R&D projects than IRAP funding.  In contrast, ITAs strongly perceive that IRAP advice is equally important to their clients—without advice, many ITAs feel that clients could not effectively assess their project development needs (financial support, technical issues, expert linkages) and could make the wrong project decisions.  The level of importance and satisfaction with IRAP advice tends to vary by client size and innovator—smaller and less innovative firms tend to place a higher value on IRAP advice than larger and more innovative firms.  In addition, clients appear to be using some of IRAP’s advisory services (technology advice, project support, referrals) much more than others (e.g., patent/ literature search, business development).  

2.1	Recommendation:  IRAP should closely examine its advisory services to increase their value to clients and their cost-effectiveness.  To do this, IRAP should consider reducing the array of advisory services available and focusing on core services used by clients; building on partnerships to complement IRAP’s core advisory services; exploring whether the type and level of advice provided should be tailored according to the specific profiles and needs of different client groups (e.g., small versus medium-sized firms); clearly defining what constitutes IRAP advice and increasing client awareness of IRAP advice as a service; and determining the optimal level of resources that should be allocated between IRAP funding and advisory services to maximize cost-effectiveness.

2.2	IRAP Response:  IRAP agrees on the importance of this issue and intends to address it over the next few years.  IRAP firmly believes that its value added advice is part of the package of services available to SMEs and difficult to separate out from the process of providing financial or other assistance.  The issue for IRAP is how to record and measure client advice activities to allow an adequate assessment of the results of advice to SMEs.  IRAP’s performance measurement framework will be instrumental in addressing this issue. 

3.0	Strengthen CTN Partnership:  The evaluation indicates that there is a continued need for helping IRAP clients improve their management capabilities.  As manager and primary funder of CTN, IRAP has had the opportunity, over the last five years, to facilitate client access to management expertise and services through CTN’s 1000 members.  CTN’s members offer advice, referrals, expertise and services to SMEs in the areas of management and technology.  As members of CTN, they have the opportunity to meet other member organizations through networking events, to communicate and seek information/ resources through CTN’s website, and to work with other members to assist SMEs.  While the evaluation indicates that CTN has provided some value to member organizations and to IRAP clients (through CTN members’ provision of management services), issues in regard to CTN’s management and leadership need to be addressed as a priority in order to better focus CTN’s activities and direction and maximize its value and benefits to CTN Members, SMEs and Canada.  

3.1	Recommendation:  CTN should apply a nationally shared and understood vision, mission and strategic and operational objectives to function as a national network, supported by a nationally shared business model that clearly articulates what CTN is about, what activities it undertakes, who its clients are, and how it is organized for reporting and accountability.  Successful regional CTN practices should be considered and CTN expectations should be effectively managed (i.e., CTN resources should be aligned with the agreed upon vision, mission, objectives and business model).

3.2	IRAP Response:  IRAP agrees that a nationally shared vision and mission are essential to CTN’s success as a national network and that this vision and mission need to be supported by a nationally shared business model that clearly articulates what CTN is about, what activities it undertakes, its resources, its reach, and the benefits to clients and to Canada.  IRAP will work with CTN Members, users and enablers of the network to complete this work by December 2003.  

4.0	Strengthen Other IRAP Partnerships:  The evaluation also indicates that clients need better access to relevant and up-to-date information and technology, and to strategic linkages (collaborators, suppliers, customers) and expertise in universities and government laboratories.  

4.1	Recommendation:  IRAP should consider improving and increasing IRAP knowledge transfer to SMEs through key partnerships with the Canada Institute for Scientific and Technical Information and other CTN members; enhancing partnerships with government laboratories and universities and building on successful IRAP relationships with NRC Institutes as a best practice; exploring opportunities to expand linkages through international networks by closely examining international networking practices established by similar programs abroad; and establishing networks within community technology clusters to enhance client access to strategic linkages.

4.2	IRAP Response:  IRAP’s new Strategic Plan identifies significant opportunities for increasing partnerships and linkages with key stakeholders to help SMEs access technologies, technical information as well as collaborations with suppliers, customers and other firms.  IRAP is currently working on implementation frameworks to take advantage of these opportunities in a timely fashion and to the benefit of innovation in SMEs.

5.0	Establish IRAP Priorities:  Over the last five years, IRAP has introduced or agreed to deliver or manage (on behalf of the Government of Canada) a significant number of new program initiatives to complement and enhance the IRAP program (IRAP-Technology Partnership Canada, Youth Initiatives, CTN, Sustainable Development Initiative, Strategic Alliances, Technology Sector Groups, Federal Partners in Technology Transfer) as well as new business processes to enhance the program’s operational effectiveness.  At the same time, IRAP has continued to deliver existing services (technological and advisory services and non-repayable contributions for R&D activities).  The evaluation results indicate that managing all of these initiatives and services has posed some important challenges.  These include ITA workload issues and limited resources and time to focus on all initiatives.

5.1	Recommendation:  The evaluation recommends that IRAP establish national priorities and effectively manage them by identifying what is IRAP’s core business and ensuring that adequate resources are dedicated to implementing and managing the core business; determining whether it is feasible to offer other initiatives based on IRAP’s available resources and selecting those initiatives based on IRAP’s new strategic plan, federal government priorities and a risk assessment; and establishing a comprehensive plan to monitor IRAP’s core business and other initiatives implemented.

5.2	IRAP Response:  IRAP agrees with the need to clearly identify priorities and will continue to do so based on resources available.  To ensure effective resource use for the benefit of SMEs, IRAP is confirming its core business, associated resources and results in developing implementation frameworks for its Strategic Plan. 
Key Operational Considerations 
The evaluation has also identified some key operational issues that should be considered by NRC and IRAP Management to enhance the program’s effectiveness.  These include adopting a more comprehensive client portfolio management approach to help maximize effectiveness of available IRAP assistance and reach clients most in need; recognizing the importance of client management capabilities and maximizing use of complimentary external resources such as CTN; and consistently collecting and maintaining client information and outcomes to more effectively measure IRAP’s performance and inform management decisions.  The following are the specific recommendations and management responses.

6.0	Adopt a more Comprehensive Client Portfolio Management Approach:  Although IRAP has succeeded to focus on technology and innovation-oriented SMEs as intended, there are other factors that should be considered in selecting clients for IRAP assistance to help maximize effectiveness of available funding/ advisory services and reach clients most in need.

6.1	Recommendation:  IRAP should consider having a portfolio that seeks an optimal mix of clients based on regular assessment of Canadian technologically-based SME needs and opportunities, IRAP client profiles, IRAP’s budget and priorities, and federal government priorities.  IRAP should strive to have a portfolio that includes attraction of new clients, and ensure that ITAs and IRAP regions have adequate capacity and responsibility to seek an optimal mix of clients (e.g., adequate financial resources, access to tools such as technology foresight and roadmaps, PPRs).

6.2	IRAP Response:  IRAP agrees with this recommendation.  As part of its Strategic Plan, IRAP is moving towards a client portfolio management approach.  Although a portfolio management approach will allow IRAP to be more proactive in terms of understanding and meeting SME needs for innovation assistance, particularly specific client communities, IRAP is driven by client demand and its client portfolio approach must reflect the reality of its business with SMEs.  IRAP is also looking to put in place the tools necessary to assist regions and staff in effectively managing and delivering within a portfolio management approach.

7.0	Recognize Importance of Client Management Capabilities:  In addition to a continued need for IRAP’s technological assistance, there is also a need for assessing and improving client business management capabilities.  IRAP needs to recognize this within the context of its technological services.  

7.1	Recommendation:  IRAP should consider increasing the program’s understanding and awareness of client management context and needs by increasing its use of CTN as a complimentary resource.  This includes enhancing ITA access to CTN business expertise and tools for assessing the socio-economic prospect of client projects and clients’ financial/ business health; and providing IRAP clients with access to CTN management expertise/ services.

7.2	IRAP Response:  IRAP is well aware of the importance for the SME of marketing and management capabilities to the successful completion of the innovation.  This has been built into IRAP’s performance framework with indicators to help assess client results in this area.  As part of its Strategic Plan, IRAP is considering how best to provide business and management advice and assistance to SMEs including how CTN might be a resource in this area.  IRAP has begun to provide training and development opportunities to its staff in areas related to business, finance and marketing and plans to continue to do so in a more systematic fashion over the next few years.

8.0	Effectively Collect and Maintain Client Data:  Collecting and maintaining client information was not done consistentIy over the last five years, and affected IRAP’s ability to demonstrate results.  IRAP has recently invested significant time and resources to establish a comprehensive information management system to collect information on clients and measure its performance.  

8.1	Recommendation:  The evaluation recommends that IRAP consistently collect and maintain data on client contacts and coordinates; the type of advice being provided to clients and on the clients receiving advice; and clients’ profiles (e.g., industry sector, size).  IRAP should also ensure that mechanisms are in place to provide adequate time and motivation for ITAs to enter information in the client management system (SONAR) and ongoing ITA awareness of IRAP’s performance measurement system.  

8.2	IRAP Response:  IRAP agrees with this recommendation and has developed a client relationship management system for this purpose.  IRAP must continually balance the need for adequate information for accountability purposes with the need for effective timely advice and services to SMEs.  This is also a resource issue, as raised in the recommendation.  Having access to client data is critical and IRAP will continue to work with staff and regions to improve access to quality information in a cost effective manner.

IRAP Overall Response:  Overall, IRAP was pleased with the results of the NRC Evaluation.  In general, IRAP agrees with the recommendations by the Evaluation for both IRAP and CTN.  In fact, IRAP’s recently approved Strategic Plan: Investing In Innovation, addresses many of the issues raised in the evaluation.  IRAP will take the results and recommendations of the NRC Evaluation into account as it proceeds with the development of its implementation plan at national and regional levels.