This report presents the results of an evaluation study prepared by BearingPoint for the Audit and Evaluation Branch and the Information Highway Applications Branch (IHAB), Industry Canada (IC).

Approach — The approach for the study involved an extensive consultation process with key informants, participants and partners in SchoolNet (SN) initiatives and activities; a review of six other previous evaluation studies of SchoolNet program components; a review of other relevant documents such as policy reports, strategic issues papers, and SchoolNet planning and research studies; and feedback on the results of the study from a SchoolNet Evaluation Steering Committee set-up by Industry Canada. The combination of the above references, including previous evaluation studies, together are based on a full spectrum of evaluation methodologies, used to garner evidence from multiple sources, including participants and non-participants in the SN program, stakeholders, successful and unsuccessful applicants, Canadian federal/provincial government officials, private sector and academic organizations, institutions, and associations. The reader is invited to consult the six other evaluation studies, referenced in this current report, since they provide an additional basis for the synthesis of findings, lessons learned and recommendations presented.

Purpose — The purpose of this evaluation is to provide input to policy and programming decisions regarding the development of the next generation of Industry Canada's Connecting Canadians and Innovation strategies, specifically as this relates to the SchoolNet family of programs. Furthermore, this evaluation contributes to addressing the government's commitment in the 2003 Federal Budget to "review all of its programs connecting Canadians to determine how best to collaborate with Canadian industry, the provinces, communities and others".

Focus on Industry Canada's SchoolNet Program — This Executive Summary provides findings and recommendations concerning the SN family of programs. It represents a synthesis of findings from this and six previous evaluation studies regarding various issues and components of the program. A roll-up of results from these previous evaluations of SN programs is integrated within the various chapters of the main report.

While the focus of this study is on Industry Canada's SchoolNet, it should be recognized that IC delivers components of this program in partnership with other federal and provincial government departments and agencies, and that these other federal and provincial departments and agencies also have separate initiatives and programs of their own that address e-learning and technology issues and needs in education environments. Responsibility and mandate for evaluating these other initiatives and programs rests with the respective other federal and provincial departments and agencies, and consequently was not included in the terms of reference for this current Industry Canada study of SchoolNet.

Evaluation Questions

The focus of this evaluation is on the following key areas of investigation that are consistent with Treasury Board Secretariat's guidelines for evaluation studies:

Relevance of the program — Is the SchoolNet program still relevant?
Achievement of objectives — To what extent has SchoolNet achieved its objectives?
Options — What are the relevant alternatives/options that would effectively address current connectivity and/or innovation challenges?
Emerging themes and challenges and the role of Industry Canada — What are the emerging connectivity and/or innovation themes/challenges for SchoolNet? What is the appropriate role of Industry Canada and what are the appropriate delivery mechanisms for SchoolNet objectives?
Lessons learned and recommendations for future policy and programming needs — What are the relevant lessons learned from SchoolNet, and what recommendations can be made for future policy and programming needs?

Findings

Continued Relevance of the SchoolNet Program

Economic and social policy outcomes — Informants consulted for this study felt that there is strong economic rationale for a SN program focused particularly on the following areas:

Building and maintaining competency and competitiveness through applications of information and communications technology (ICT) in education.
Moving from connectivity to innovative applications — to build wealth through knowledge and a sense of community.
Achieving a flexible workforce through a system of learning that re-trains and re-educates on an ongoing basis — i.e., a lifelong learning system.
Developing a competency-based learning system that is responsive to the economy and the needs of industry.
Promoting innovative project-based solutions for learning.
Fostering partnerships that traverse political and cultural boundaries, and engage industry and educational institutions

There is also strong rationale presented by key informants for a SN program focused on social outcomes — namely, to contribute to narrowing the "digital divide" in education, focusing on educational institutions in rural and remote regions of the country, First Nations schools, and schools in economically lagging urban locations. A focus on disabled students and students at risk are also areas of social concern that SN could potentially address.
Targeting the program — There is strong endorsement by key informants that the SN program is still relevant, and that it continues to evolve and support sophisticated users of ICT in education — i.e., the early adopters and the innovators. However, there is also consensus on addressing the needs of late adopters and uninitiated users, to the extent that resources permit. In addition, while K-12 students are primary targets for SN, post-secondary and adult training is considered a high priority, especially by industry and post-secondary institutions. The latter is seen as a more immediate priority, while the former is seen as a medium to long-term priority.
Consolidating SchoolNet — Over the past decade, SN has played a leadership role in addressing connectivity and ICT capacity issues. Some of the key informants for this study have suggested that the program now needs a new focus and programming that converges the rationale of the program around the key economic and social policy themes identified above.

Objectives Achievement

The connectivity objective of the SN program has been achieved (15,300 schools, including 480 First Nations schools and 3,400 libraries have been connected and more than 450,000 refurbished computers have been provided to schools and libraries). However, technology upgrades and ICT capacity building continues to be an ongoing mission, requiring long term commitment by all those involved in the education system of Canada, with the support of government and other participants and partners.
The SN program is credited by most of those interviewed for this study, and as evidenced in previous studies, as having resulted in a major shift in teacher's skills in the use of ICT to enhance educational opportunities. This is consistent with the recent Canadian Teachers' Federation Report which found that most teachers have accepted computers and other information and communication technology in their classes and are integrating ICT in their lessons and concluded that "eight out of 10 teachers say computers are essential or important in the way they teach (three out of 10 say essential)."
The SN program has provided an effective and broad forum for the exchange of ideas and debate among educators, and for sharing knowledge and success stories in the use of ICT for educational purposes. This apparently has had a spin-off effect on late adopters of technology in the education community.
The program has empowered teachers, empowered students, and provided significant value for money incentives to migrate to the use of ICT in the classrooms and other learning environments in Canada.
The program has provided new learning opportunities through technology projects (e.g., through 29,400 GrassRoots classroom projects) that have been scoped broadly across regional, national and international levels. Through collaborations involving various school projects, the program has brought students and teachers together in a broad spectrum of innovative applications.
Initial program recipients have had a cascading effect on others to take advantage of the program, creating a demand and a culture of change among some schools within Canada (e.g., 150 Network of Innovative schools).
It is generally recognized by those interviewed for this study, and based on evidence from previous studies, that without SN there would be far less collaboration and development in the use of ICT in the Canadian educational system today. Further, it is considered to be highly likely that with any overall roll back or abridgement of SN initiatives there would be a reversal in Canadian learning environments in the use of ICT for education, largely due to limited resources in provincial education budgets and other competing priorities of the system.
Partnerships enabled through SN initiatives have been a very successful program delivery approach. Key informants, and the evidence from previous studies, suggest that thousands of collaborative and partnership initiatives have resulted from SN activities, since the inception of the program.
Building ICT capacity in First Nations schools is an unfinished job. A majority of teachers in First Nations still do not have the requisite skills and experience to implement ICT into their teaching.
There is a consensus among educators interviewed that SN program managers and administrators at IHAB have done an excellent job in delivering a complex program, and in designing program instruments for education that address challenging technology issues of national concern. On the other hand, while the original SN program targets were effective for program administration and delivery purposes, there is now a general impression that SN has less clarity about its targets and purpose.

Options

Continuing to support the early adopters and innovators in future program design and delivery — An underlying principle of the SN program has been to support early technology adopters and innovators in the educational arena. This principle is seen as having continuing relevance to current and future challenges in program delivery.
Teaching the teachers — Program design mechanisms that target teachers to help them apply ICT are deemed necessary (e.g., teacher mentoring programs, networking, and best practices tools and repositories). Teaching the teachers, however, is primarily a provincial responsibility, and Industry Canada can only contribute to this effort indirectly as a partner to the provinces and local schools and school boards.
Supporting the development of learning objects and standards — There is a need to continue to support the building of learning objects by establishing a repository for these objects. The program could also pursue a nationally coordinated initiative towards the adoption of voluntary standards in the application of ICT to education — including standards for learning objects, quality, and interoperability specifications for e-learning.
Narrowing the digital divide by improving access — Equity of access to ICT in learning environments is a social as well as an economic goal. The SN program has addressed this goal over the years (e.g., First Nations program), but increased efforts and collaborations with key responsibility centres in other federal, provincial or municipal governments are needed to adequately address increasing "digital divide" challenges — particularly for rural and remote regions.
Research to gauge private sector needs and interests, and to measure ICT impacts on learning — Program options and delivery mechanisms need to be devised that are based on a sound understanding of business models that are likely to engage private sector participants in future program initiatives. New research and market analysis is needed with a focus on developing appropriate mechanisms to accomplish this goal. The SN program could also contribute to research towards the ongoing challenge of measuring the real impacts of ICT on learning.
Implementing community-based program options — Other Industry Canada programs have focused on community-based initiatives (e.g., the CAP program). In this respect, SN has an opportunity to devise program mechanisms that encourage innovative community-based technology applications for local learning institutions.

Emerging Themes and Challenges

From its inception the SN program has relied extensively on partnerships and collaborations with provincial and territorial governments, school boards and schools, post-secondary institutions, educational associations, volunteers, and the private sector. An earlier evaluation of the program (December 2000) concluded that the program has been "outstandingly successful" in partnerships and that "the wide range of SN partnerships represents one of the key successes of the program". On the other hand, successes of the program have varied across provinces, with some provinces opting out of some SN partnership initiatives and program components due to different socio-political dynamics and prevalent education systems and jurisdictional issues outside the federal government's control.
The current situation, however, suggests that there are continuing and new partnership and collaborative opportunities emerging for the SN program in the following areas:

Developing standards and accreditation requirements in educational applications of ICT.
Mutual recognition of credits across educational institutions.
Community-based initiatives, broadening the scope of educational opportunities for K-12, adult education, skills learning, and on the job training.
Developing and supporting campus consortia that foster the development of online applications for post-secondary education and lifelong learning programs.
International collaborations focused on sharing educational content and networking opportunities.

There is also a need to re-engage the private sector, including big players and SMEs, as originally done by the program in its earlier manifestations. With experience, SN rapidly learned that partnerships with the private sector were desirable in themselves, since they helped build participation and sustainability — particularly for program components such as Computers for Schools (CFS), Learnware and SchoolNet Youth Employment Strategy funded programs.
There is general recognition that there are benefits to evolving community-based initiatives, but that these initiatives also need better broad-based educational tools, and building blocks to support local initiatives. This includes the development and application of educational portals, training modules, and learning objects.
The contributions of the SN National Advisory Board have been instrumental to the success over the years of the SN program — particularly in providing policy and strategic input to help shape the direction and focus of the various SN components. There is a need, however, to re-confirm the role and effectiveness of the Board, in light of emerging partnership challenges, provincial priorities, and the social and economic rationale underlying the SN program.
The Innovation Agenda of Industry Canada is a central theme that continues to guide program initiatives across the department. For the SN program, "innovation" has focused on new applications of ICT to promote educational opportunities in the classroom and other learning environments (virtual or otherwise). Key informants suggest that in going forward with the SN program, there is a need to clarify what "innovation" means in the context of SN, particularly in light of continuing and emerging technology themes and challenges.
These emerging technology themes and challenges involve the following:

Broadband application.
Learning objects and interoperability specifications and standards for e-learning.
Building repositories of knowledge and contemporary tools for ICT applications.
Rich multimedia.
Floating infrastructure (e.g., wireless, laptops) versus fixed infrastructure (e.g., desktops).
SN program support for R&D (e.g., for learnware applications)
Recycling (involving environmental issues) and re-using computers (involving supplying computers for schools and libraries).

The SN program over the years has built up and maintained a high profile, with visibility among educators in this country, and abroad. As some key informants have suggested this "brand name" is an asset that needs to be capitalized on in future configurations of the SN program.
However, there is also concern expressed that the program has become too "fragmented" in that it has diversified into many sub-themes, to the extent that there is now a need for a new "man on the moon" vision for the program — to consolidate its purpose and strategic directions over the next few years.
The program has achieved significant clarity with the issuance of its Results-based Management and Accountability Framework (RMAF) (most recently updated in February 2003), and its Risk-based Audit Framework (RBAF) (most recently updated in June 2003). These documents are comprehensive and represent a significant achievement in establishing appropriate structures for program performance and financial accountability. The challenge remains, however, to establish an effective and ongoing information gathering mechanism that feeds into the reporting scheme of program management.
In this respect, several program evaluations have been successfully concluded over recent years. These evaluations have identified the achievements and key issues involving the SN program components. The responses of IHAB to the issues raised by these evaluations have been appropriate and comprehensive, providing additional rationale and mitigation strategies to address risks or misconceptions about program intents and delivery mechanisms.
However, there is still a need to further the state of the art in measuring impacts of ICT in learning situations. The jury is still out on the extent and character of ICT impacts, the specific indicators that best gauge these impacts, and the ultimate effects of an ICT enabled education system on the Canadian economy and the social wellbeing of Canadians.

Role of Industry Canada

The SN program has stayed well within the connectivity/innovation mandate of Industry Canada, with a clear focus on technology applications and the role of ICT in progressing Canadians towards an economy with rapidly changing skills and knowledge requirements.
Focusing efforts on connectivity is a continuing and relevant goal for Industry Canada and for SN. However, interviewees in this study generally agree that there is a more complex set of issues that need to be addressed by IHAB and Industry Canada, including the strategic use of e-learning technology, in a very broad sense, for preparing the current and future Canadian workforce for the evolving knowledge-based economy. This broadened scope includes a complete and holistic view of learning as a lifelong endeavour — from K-12, to post-secondary, to on-the-job training, and adult learning.
The evidence from past evaluation studies, and from the majority of interviewees consulted for this study, agree that IC is in a good leadership position to articulate a clear vision of where Canada needs to be in five years with regards to the application of ICT to education, and to clarify the actionable goals and program designs to get there.
The infrastructure and network of partnerships that SN has built over the years validates this view, and corroborates the likelihood of a successful re-deployment of the program during, and beyond, 2004.
The consensus view is that without IC's leadership through SN, in evolving the use of ICT for education and learning in Canada, progress on this front will be slowed. The consequences would be a fracturing of initiatives, and disparities in progress across Canada and between provinces and territories.

Lessons Learned and Recommendations

The lessons learned and recommendations presented in this section of the Executive Summary represent a synthesis of the results from consultations and analysis of this and six previous evaluation studies of SchoolNet program components. The recommendations presented cover a broad policy and program spectrum, based on feedback and evidence from multiple sources. This evaluation report is intended to provide one source, amongst others, that informs future policy and program priorities, in response to lessons learned and recommendations emanating from the evaluation work undertaken. It is the prerogative and responsibility of Industry Canada and the Information Highway Applications Branch to provide an appropriate management response to the findings and recommendations, and to establish priorities, for future connecting Canadians program initiatives.

Flexibility of the program — The SN program is generally deemed to be a flexible program delivery mechanism in that it has relied on building extensive participation of partners, and in that it has been successful in responding to various opportunities in different jurisdictions, within the bounds of IC's mandate as a catalyst and facilitator of innovative technology applications, and within the context of the departmental agenda for progressing Canadians towards an economy with rapidly changing skills and knowledge requirements. The Canadian education system varies from province to province, and the flexibility of the SN program in responding to these differences has served its purpose well by allowing it to adapt to changing needs and situations between and within provincial/territorial jurisdictions.
Recommendation: For future programming design needs, it is recommended that SN retain a flexible delivery approach that responds to the different requirements of the varying provincial/territorial and First Nations education systems across Canada.
Mission — SN was served well by an original, consensus-building long-term connectivity goal for implementing ICT in schools. There is currently a need to articulate a similar, consensus-building steady and long-term mission for SN. In other words, there is a need to clarify the role of SN as it relates to where Canadian education should be five years from now with respect to the use of ICT.
Recommendation: The SchoolNet National Advisory Board could be recruited for this purpose — with a special working group from within SNAB commissioned to address specific challenges and workable solutions. The "Foresight" document prepared by SNAB in this respect is an appropriate foundation to build on. Active representation on this working group from educational institutions, private sector stakeholders, and federal and provincial/territorial government departments is required.
Program components — The GrassRoots and Network of Innovative Schools (NIS) program components have proven to be successful program models for delivering ICT to the classroom — specifically referring to the project-based program support design and concept (GrassRoots) and the professional teacher development aspects (NIS) underlying these components.
Recommendation: Future program designs of SN should continue to incorporate project-based delivery and professional teacher development schemes as a means to effectively engage partners and to implement innovative ICT solutions for education.
Partnerships and voluntary participation — The partnerships and voluntary nature of the many SN initiatives has had very strong appeal to educators over the past decade. SN was not a compulsory program for any participant, and was not built into a curriculum-based framework. This character of the program enabled active partnerships and participation of provincial, municipal and other federal government departments, as well as volunteer groups, school boards, and private sector organizations. The recent RBAF for SchoolNet (June 2003) stipulates that the number one risk associated with the program is "the loss of partners and availability of volunteers" for which various and ongoing mitigating strategies have been developed and implemented.
Recommendation — SN should continue to implement its stated goals through program initiatives that are based on voluntary participation of partners, providing value-for-money financial and in-kind incentives that contribute to the adoption and use of ICT by educational institutions and learning environments.
Program alternatives/options — Continued federal government support for the proliferation of ICT for learning through supply program delivery mechanisms (such as Computers for Schools) and capacity-building programs (such as GrassRoots and NIS), may not be sustainable in the long-term. This is particularly the case in times of resource constraints and alternate government priorities, and when dispersion of federal-provincial government tax dollars is involved. Therefore, identifying options for sharing most cost and outcome effective practices that respond to particular requirements of provincial/territorial and First Nations education systems is needed, and is a responsible initiative to undertake.
Recommendation: A study of the most cost and outcome effective practices for different provincial and territorial education systems, will contribute to determining if there are appropriate alternative delivery mechanisms that meet present and future challenges. In addition, cost-effectiveness comparisons of Canada's SchoolNet program to other national SchoolNet programs in other countries would provide valuable insights. Industry Canada should initiate a comparison study to identify the most cost and outcome effective practices for continuing to integrate information and communications technology in learning environments.
Innovation — Fostering innovative applications in education has been a main goal of the SN program. However, what is considered to be innovative has evolved with the advent of new and more robust technologies, including wireless and rich media. There is a need to redefine what is considered innovative and what is now state-of-the-art (e.g., the design and proliferation of web pages in itself is hardly an innovative raison d'être for SN any more).
Recommendation: Any new SN policy and program design, that focuses on early adopters and innovators, needs to "raise the bar" on what constitutes innovation. Naturally, the relevance and impact of innovative initiatives also need to be considered.
Entry levels — The ICT competency levels of educators and students varies across Canada, within provinces, schools, and even classrooms.
Recommendation: Future SN program designs need to consider different entry levels for participants in the program—e.g., based on different skill levels, age groups, geographical distributions, cultural differences, and provincial/territorial school systems and characteristics.
Networking opportunities — Educators see opportunities for networking, created by SN program activities and events, as a major benefit of the program. Without the networking opportunities created by SN many successes in ICT applications would not have otherwise occurred.
Recommendation: SN should plan and budget for frequent networking events (both virtual and face-to-face) to bring together participants and partners in the program, particularly educators, to learn from each other and share results of their initiatives.
Regionalization and devolution — The regionalization and the devolution of SN program initiatives to local responsibility centres are considered by program participants as a positive characteristic of the program. This, for example, is considered a very strong point of the First Nations program.
Recommendation: Industry Canada and the SN program should continue to rely on a decentralized delivery mechanism for its program components, while retaining overall budget control and oversight responsibilities, as well as policy and program design and delivery decisions.
Collaboration with schools and school boards — The SN program overall has received high marks from schoolteachers and principals, and school board representatives across Canada.
Recommendation: Collaboration with schools and school boards is essential, if the program is to continue to succeed.
Role of Industry Canada and the private sector — The role of Industry Canada as a promoter of ICT for education is valid and should continue, but a current gap is the absence of drive to foster partnerships with the private sector.
Recommendation: Research should be supported to understand the appropriate business model and incentives to engage the private sector in future SN initiatives.
Issues scan — The timing seems right to do a broad issues scan of what is needed and what needs to be done for future applications of ICT in education in Canada.
Recommendation: Industry Canada is in a position to provide leadership to oversee such an issues scan, and therefore should consider undertaking this initiative.
Absence of a national strategy for ICT in education — Results from SN initiatives often evolve slowly and the process of fostering innovation and innovative uses of ICT in education requires a long-term commitment, by all stakeholders, and by federal/provincial governments. In the absence of a national strategy for the use of ICT in education, it is very important to at least articulate a clear vision that creates consensus for action.
Recommendation: Industry Canada, with the participation of the SchoolNet National Advisory Board, should play a leadership role in articulating this vision.

Final Report (PDF - 459KB - 63 pages)

Management Response - January 2004 (PDF - 96KB - 8 pages)

Implementation of the Management Response Update to the 2004 SchoolNet Evaluation - June 2005 (PDF - 35KB - 10 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.