Summative Evaluation of CANARIE Phase 3
Audit and Evaluation Branch
Industry Canada
December 2003
Executive Summary
Introduction
CANARIE is now in the fifth and final year of Phase 3 (1999-04) of its three-phase program, originally approved in 1993. Phase 3 began on April 1, 1999 and is scheduled to be completed by March 31, 2004. CANARIE received $78 million from the federal government over the five-year period on a cost shared basis. The objective of Phase 3 is:
“…catalyze the development and diffusion of technologies, applications and services that may underpin and/or use advanced Internet infrastructure, such as CA*net 3 and CA*net 4, and are based on open-systems standards.”
In support of this objective, CANARIE has one program, the Advanced Applications Development Program (AADP), which is supporting research and development by industry and the public sector in developing and deploying advanced applications in the sectors of e-business, e-learning, e-health and intelligent systems. The intelligent systems program is managed by Precarn under an arrangement with CANARIE. There are also a number of other programs operated by CANARIE outside the Phase 3 agreement.
The objective of this study is to carry out a summative evaluation focusing on the evaluation issues of relevance, success, and future directions for CANARIE after Phase 3. The element of cost effectiveness is being covered in a separate study of CANARIE. The intelligent systems program managed by Precarn was evaluated in a similar manner to the programs operated by CANARIE. The evaluation study also took into account the possible extension or reprofiling of Phase 3 to March 31, 2005.
Program Profile
The AADP represents a shift in focus away from the network infrastructure development of the previous two phases towards the development of advanced network applications and related technologies. This shift reflects both the successes of the previous two phases and the need for Canada to develop an advanced network applications industry that will help to brand Canada internationally as a high-tech advanced economy that can compete successfully with the United States, Europe, Japan and other leading countries. As noted, AADP is supporting projects in four application areas: e-business, e-learning, e-health and intelligent systems.
Phase 3 activities are oriented toward assisting key sectors of the economy to develop critical advanced applications and associated infrastructure that will improve their structural competitiveness. It was expected that the applications would utilize advanced networks including CA*net 3, and more recently CA*net 4, developed through the Next Generation Networks program, together with the regional networks that connect to it.
The AADP provides financial assistance of up to 50% of eligible project costs. Of the program's $78 million budget, $28.4 million has been allocated to the e-business program, $28.4 million to the e-learning program, $9 million to the intelligent systems program (Precarn) and $4.5 million to e-health. The remaining $7.8 million or 10% of the program's budget is earmarked for project management and administration costs of CANARIE and Precarn.
Study Approach
The evaluation study involved four lines of enquiry: document reviews, consultations, a survey, and case studies, and was guided by the evaluation framework prepared in March 2000 and the interim evaluation of November 2001. A breakdown of the consultations (49 planned/48 completed), survey (72 contacted/33 responded) and case studies (3) is provided in the following table.
Summary of Interviews, Survey and Case Studies
| Enquiry Groups |
Number Planned & Method of Enquiry |
Number Completed / Responded |
| 2 interviews |
2 interviews |
| 2 interviews |
3 interviews |
| 4 interviews |
4 interviews |
| 3 interviews |
3 interviews |
| 3 interviews |
3 interviews |
| 1 case study |
1 case study |
| 10 interviews |
9 interviews |
| 20 surveyed |
9 respondents |
| 1 case study |
1 case study |
| 10 interviews |
7 interviews |
| 22 surveyed |
14 respondents |
| 4 interviews |
5 interviews |
| 21 surveyed |
6 respondents |
| 1 case study |
1 case study |
| 6 interviews |
7 interviews |
| 9 surveyed |
4 respondents |
| 2 interviews |
3 interviews |
| 3 interviews |
2 interviews |
Study Findings
Program Relevance
The objectives of Phase 3 continue to be relevant, especially given the goals and targets outlined in “Canada's Innovation Strategy”. The target of moving from 15th to 5th in international rankings of R&D; performance will require the infrastructure and applications development that CANARIE has been initiating and supporting in Phase 3 and previous phases. CANARIE is seen by many as an organization attuned to the needs of the time, able to evolve to meet changing needs. In Phase 1, its focus was on technology and product development; in Phase 2, attention moved to incorporate applications; and in Phase 3, the program has concentrated on sectoral applications. The importance to Canada of CANARIE's work on infrastructure development was also emphasized and, while outside the scope of this evaluation, we noted comments such as “without CANARIE, Canada would not have a high-speed research infrastructure which has been essential in keeping Canada in the forefront of research in areas like genomics, and high energy physics involving the transfer of large volumes of data”.
Further, many believe that CANARIE is more necessary today than at the start of Phase 3 in 1999. With the recent dot-com crash, venture capitalists have become extremely conservative, and the ability of companies to raise financing has become tougher. Project approval from CANARIE or Precarn, however, has reduced risk and allowed companies to leverage additional sources of financing. Venture capitalists and the e-business, e-learning, e-health or intelligent systems communities know that if a project has been approved by CANARIE or Precarn, it has been subjected to rigorous review by peers, and has attained a certain standard, or “seal of approval”.
Both CANARIE and Precarn indicated that both the number and quality of proposals increased over the 1999-2003 period. During this time, 517 Expressions of Interest (EOIs) were submitted to CANARIE, of which only 134 were accepted for the proposal stage. Some 87 proposals were recommended for funding, giving an approval rate of 17 percent of EOIs received, and 70 percent of subsequent proposals received. In the same period, Precarn funded 34 of 92 proposals for an approval rate of 37 percent.
Success/Objectives Achievement
The table below summarizes the number of projects by program area, total project costs, the contribution by CANARIE, and the leveraged contributions.
Summary of Phase 3 Contributions
| Program |
No. of Projects |
Project Size
(000,000) |
Phase 3 Contribution
(000,000) |
Leveraged Contributions
(000,000) |
Leveraged Contributions
Percent |
| E-Business |
29 |
$65.6 |
$28.3 |
$37.3 |
57% |
| E-Learning |
33 |
$64.0 |
$30.0 |
$33.9 |
53% |
| E-Health |
23 |
$11.8 |
$4.6 |
$7.2 |
61% |
Sub-Total:
CANARIE |
85 |
$141.3 |
$62.9 |
$78.4 |
55% |
Intelligent
Systems |
13 |
$31.8 |
$9.0 |
$22.8 |
72% |
| Total: Phase 3 |
98 |
$173.2 |
$71.9 |
$101.3 |
58% |
Summary of Phase 3 Contributions
Phase 3 is not at a point where it can demonstrate final results although the evidence from the interviews, survey, and case studies suggests that the program, despite initial delays, is on course to achieving the intended impacts and effects. It should be noted, however, that an audit/evaluation is conducted at the end of each project, which will provide Industry Canada with additional information on impacts. In e-business, e-health, e-learning, and intelligent systems, a major achievement of the program has been the promotion and facilitation of collaboration among organizations and individuals in creating network applications. A key impact has been the development of communities of interest, and full involvement by SMEs, particularly in e-business projects. CANARIE exercised considerable care in choosing the communities of interest to be supported and involving the appropriate industry associations. A critical challenge in the four areas has been the social and cultural issues around communities of interest.
In e-health, the major impact has been on demonstrating the unique contribution telehealth can make to improving the quality of health care services. However, until the provinces agree to implement and integrate telehealth applications into the delivery of health care, the full impact of the e-health program will not be realized. In e-learning, Canada is leading the world in learning object repositories but not in implementation of e-learning initiatives. In intelligent systems, the major impact has been to move Canada further up the innovation chain, as well as the development of communities of interest and the greater participation of SMEs. All participants in this program believe that intelligent systems will play an increasingly important role in the Canadian economy, particularly in the space, mining, biotechnology, health care, and geomatics fields.
Most project participants that were interviewed indicated they had met the objectives of their projects. Delays in the “go to market” phase were the main reason for not fully achieving objectives. The survey confirmed the high level of satisfaction of participants with the CANARIE program: meeting project needs (72.7 percent); enhancing the quality of products/services (75.8 percent); developing new products/services (67.7 percent); developing new partnerships (75.0 percent); increasing revenues (33.3 percent); increasing the number of high quality jobs (57.6 percent); increasing the uptake of network applications (45.5 percent); and, changing service delivery costs (30.3 percent).
Network Usage by Projects
CA*net 4, as did its predecessor CA*net 3, interconnects the provincial research networks, and through them universities, research centres, government research laboratories, schools, and other eligible sites, both with each other and with international peer networks. The use of CA*net by the application programs (e-business, e-learning, e-health) corresponds closely to access. Under Phase 3, 55 percent (18 of 33) of e-learning projects, and 22 percent (5 of 23 projects) of e-health projects used CA*net to carry out their research, while none of the e-business projects used CA*net. E-learning and e-health projects where the lead participant was a private firm, also did not use CA*net, the main reason being lack of, or difficulty in getting, access to CA*net by the private sector partners. In e-business, another reason for not using CA*net has been that the applications being developed have not required the large bandwidth of CA*net; they have required innovative network management but have been readily accommodated within the capacity of the publicly available Internet. It is interesting to note that in e-business, while none are using CA*net, 24 percent (7 of 29) of the projects are using wireless networks.
Future Directions
All respondents to the survey and interviews indicated that federal support of advanced network applications and content development (e-business, e-learning, e-health, and intelligent systems) should continue. Momentum has been created over the ten years of CANARIE - the results of investment in Phases 1 and 2 are beginning to be evident in the work of Phase 3 — but the task is not considered to be complete. A general consensus in our data gathering is that CANARIE and Precarn have been key factors in helping Canada to maintain its competitive edge in network applications with our major trading partners, e.g., USA, Europe, Japan, Australia. In the absence of these programs, most believe industry in those countries would move ahead of Canada, especially given the relative generosity of their R&D; programs compared to Canada.
A common view held by industry is that a future focus has to be the mid-market, that is SMEs. The software approach to e-business has not worked for SMEs as indicated in Chapter 5 because SMEs haven't the necessary skills to adopt the solutions or are slow to adopt, and, furthermore, they can't afford them. The respondents advocated an honest broker role working with industry associations and suppliers to develop a 'communities of interest' approach. It was felt that CANARIE had made a start in this direction in Phase 3 and had the knowledge, experience and credibility to continue on this path.
Also of interest in considering future directions is the issue of Canada's innovation challenge. Part of this challenge is seen as the acceleration of research collaboration, for example between research laboratories, between industry, university or government. “Data sharing is now pushing the frontiers of science. Collaboration is needed but it poses a challenge – all sectors are saying we need facilitation, cross fertilization, informed advice and leadership.” Again, this was seen as a role that CANARIE has played and could play to a greater degree in the future.
The approach to collaboration and to programming in network applications could, however, change in the future. In order to gain access to global markets, collaboration will need to be international in scope. A comment by an industry interviewee summarized the future situation: “Collaboration, consultation and facilitation will continue to be needed in the future supported by public funding, whether or not CANARIE is there as a performer of these functions”. As network processes become more integrated, there would also be benefit, according to some interviewees, for the now separate areas of e-business, e-health, and e-learning to be brought together into one program.
The results of the survey provide a breakdown of support (primary, secondary) for various possible roles for CANARIE. Continued funding of advanced network applications and maintenance of CANARIE's coordinating/facilitation (honest broker) role received strong endorsement (primary and secondary). Across the application areas, the top preference among respondents for funding was basic R&D;, with the development of prototypes a close second.
Most respondents noted that if there is a Phase 4, part of the funds should be used to finance demonstrations for the purpose of showcasing the advantages and benefits of CANARIE and Precarn. Many thought that a big challenge facing both CANARIE and Precarn is raising awareness, not only to potential project participants but also to the Canadian public. The general consensus was that stronger promotional efforts might have made the future of CANARIE and Precarn more certain. Several applicants (particularly in e-learning and e-health) whose proposals were not funded, expressed a desire to see detailed results of completed projects. They noted that the information provided in the newsletters is of a general nature, and although very useful for informing people on the types of projects completed, does not provide the detailed information that they would need to utilize and apply the results of CANARIE-funded research to their R&D; efforts.
Conclusions and Recommendations
The following section summarizes the major conclusions that we have drawn from our research, and are grouped into two main categories: first, the possible extension of Phase 3 for one year, including the issues of relevance and success; and second, possible future directions for CANARIE after Phase 3. It should be noted that the scope of the evaluation was restricted to the Advanced Applications Development Program (AADP) and specifically the application programs of e-business, e-health, e-learning and intelligent systems (managed by Precarn). The study's mandate did not include CA*net or other application programs, such as e-content, that are supported by other departments and royalties from Phase 2.
Extension of Phase 3
CANARIE has requested an extension of Phase 3 to March 31, 2005. The study was asked to include this request in its review of Phase 3.
Conclusions
We conclude, based on our research, that there is merit in extending Phase 3 to permit all AADP projects to be completed. Supporting this position is our conclusion that the objectives of Phase 3 continue to be relevant, especially given the goals and targets outlined in “Canada's Innovation Strategy”, and that CANARIE is meeting the high-speed infrastructure and advanced network application R&D; needs of educators, health practitioners and private firms. Although Phase 3 is not at a point where it can demonstrate final results, due in part to the delay in getting the AADP started, evidence from completed and ongoing projects indicates that the AADP is on course to achieving the intended impacts and effects. Industry Canada's support for AADP has put many of the participants at the forefront in their respective sectors.
Only 38/85 projects are or will be completed by March 31, 2004, with the remaining 47 projects being completed six months later by September 30, 2004. All 13 of the intelligent systems projects under AADP managed by Precarn are or will be completed by March 31, 2004. The termination of the AADP on March 31, 2004 would mean, therefore, that 47 projects would remain unfinished or possibly continue at a reduced pace without CANARIE funding. Project objectives would not be reached and potential follow-on service implementation on an operational scale or product commercialization would likely not happen.
Recommendation
CANARIE's request to extend the AADP to March 31, 2005 should be approved so that the remaining 47 AADP projects can be completed, and the intended impacts fully realized.
Future Directions
CANARIE is now in its fifth and final year (1999-04) of its three-phase program. Industry Canada will need to decide if it wishes to keep to the original Phase 3 sunset, or to support a subsequent phase. A factor will be the perceived need for continued federal funding and the future role that might be played by CANARIE. We outline below a number of conclusions from the study and options for the future.
Conclusions
The technologies that have been supported in the AADP projects continue to evolve and are providing opportunities for applications that will involve new uses of advanced networks. The key change taking place will be the integration of the Internet into applications such that the applications can take advantage of other data and computational resources that they can directly access over the network. CANARIE, Precarn and others recognize that significant technological barriers need to be overcome as well as organizational and cultural barriers to allow this transformation to happen.
Our evidence is that a facilitator at a national level will be needed if Canada is to be able to take full advantage of these new applications that could have a profound influence on business practices, the Canadian health care system, the learning environment, and the manufacturing and resource industries. CANARIE has performed this function in the past, particularly in Phase 3, and has been credited with overcoming institutional and cross-jurisdictional impediments in a way that government agencies would have found difficult to do. All participants and other interviewees (including one rejected applicant) were in favour of a subsequent phase. Consultation with a provincial and US representative familiar with CANARIE also recognized the key contribution being made by CANARIE in advanced network development and utilization.
The leverage of funding from other sources that the AADP has achieved in Phase 3 is notable, particularly for the Precarn managed component. This reveals a strong commitment by firms and other organizations in the private and public sectors to work with CANARIE and Precarn in a coordinated effort to capitalize on the potential of new network and system technologies and applications. The evolution of CANARIE's programs and the broadening of its stakeholder base through Phases 1-3 also has demonstrated a capacity on the part of CANARIE to change to ensure Canada stays on the leading edge of network technology advances and the opportunities afforded to raise the productivity and competitiveness of Canadian companies and public services.
The four application areas of the AADP can be divided into those with a public sector focus, e-learning and e-health, and those with a private sector focus, e-business and intelligent systems, both having distinct characteristics that call for a differentiation in their requirements for support. This we believe to be a significant finding in considering future support in these areas.
- The study results indicate that the key determinants in supporting public sector application areas are: 1) projects would not proceed without public support; 2) cost sharing of the project creates significant difficulties; 3) participants' main interest is financial support for basic R&D;; 4) most participants have access to CA*net and are using the network to carry out their research; 5) most do not expect to commercialize and/or pay royalties; 6) projects are dependent on the provinces for deploying the results of their project; and 7) large broadband connectivity such as CA*net remains an issue for the K-12 levels in e-learning and for rural and remote communities in e-health.
- By contrast, participants in the private sector programs noted that: 1) a primary need is financial support for pre-commercial testing activities and development of prototypes; 2) sharing of project costs with CANARIE/Precarn is fair, and in some cases desirable in facilitating the commercialization of results; 3) acceptance that failure to come up with their contribution means the project should not be funded; 4) without CANARIE/Precarn funds, projects would proceed at a slower pace and with a smaller scope; 5) none have access to CA*net and are relying on publicly available internet to carry out their research; 6) most expect to commercialize the results of their project and/or pay royalties; 7) projects are dependent on venture capitalists and other private funding organizations for deploying the results of their project; 8) their applications do not require the large broadband of CA*net; and 9) for e-business projects, the development of communities of interest is significantly more critical to the success of the venture than in the case of intelligent systems projects.
Recommendation
Consideration should be given to providing federal funding for a subsequent phase of CANARIE, subject to an acceptable proposal from CANARIE that reflected the key findings of this study, namely, that:
- The communities of interest and communities of practice approach initiated in Phase 3 projects be continued and enhanced, thereby expanding CANARIE's stakeholder base;
- The involvement of SMEs continue to be a focus of attention;
- The different funding, commercialization, royalty payment, and network requirements of public and private sector application programs be recognized in program design and management; and
- Resources be allocated to publicizing through demonstrations the applications under development in order to broaden awareness of the potential benefits of the applications and facilitate their implementation.
Final Report (PDF - 480KB - 58 pages)
Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.