2004-613 Evaluation Framework for the Government On-line (GOL) Initiative, Final Report

October 27, 2005

• Executive Summary
  
  
• 1.0 Introduction
  • 1.1 Background and History
  • 1.2 GOL Governance Structure
  • 1.3 Purpose and Scope of the GOL Evaluation Framework
  • 1.4 Framework Development Methodology
  
  
• 2.0 Component Profile of GOL Initiative
  • 2.1 Description of Components
  • 2.2 Stakeholders
  • 2.3 Logic Model
  • 2.4 Activities to Outcomes
  • 2.5 GOL Funding
  
  
• 3.0 Understanding the GOL Context
• 4.0 Evaluation Issues
• 5.0 Approach to Cost Effectiveness
• 6.0 Evaluation Options
• 7.0 Conclusion
• 8.0 Appendices
  • Appendix A: Mapping GOL Strategic Objectives Year-Over-Year
  • Appendix B: Concordance Between PMAF Dimensions/Indicators and the Evaluation Issues
  • Appendix C: Approved GOL funding
  • Appendix D: Principal Documents Consulted
  • Appendix E: List of Individuals Who Contributed
  • Appendix F: Evaluation Options – Costing by Data Source
  • Appendix G: Methodology for the Measurement of GOL Cost-Effectiveness
  • Appendix H: Sample Departmental Survey

Executive Summary

This Report has been prepared by the Audit and Ethics Branch (AEB) with the assistance of Consulting and Audit Canada (CAC) in response to a request from the Office of the Chief Executive Officer (CEO) of Information Technology and Systems Branch (ITSB), for the development of an Evaluation Framework for the Government On-Line Initiative (GOL).

GOL was conceived to establish the foundation for e-government in all Canadian government organizations by targeting key technological, policy and service delivery advances that could be rapidly shared and implemented by federal organizations and organizations of other government jurisdictions throughout Canada thus ensuring better value on technological investments.

GOL was originally supported and managed through the Chief Information Officer Branch (CIOB) of the Treasury Board Secretariat (TBS). On December 12, 2003, the operational management of the GOL Initiative was transferred to the Information Technology Services Branch (ITSB) at Public Works and Government Services Canada (PWGSC) while responsibilities for the policy side of GOL remained with the Treasury Board of Canada Secretariat. The GOL components such as Gateways and Clusters and the Secure Channel, which had evolved past the design and feasibility phase into an implementation phase were also assigned to ITSB. In December 2004, responsibility for the implementation of Gateways and Clusters was subsequently transferred to the Government Information Services Branch (GISB) within PWGSC.

The following key, senior level committees have been active in the governance and direction of the GOL Initiative:

1. TBSAC Information Management Sub-Committee of Deputy Ministers (TIMS);
2. Service Management Board (SMB); and the
3. GOL Leads.

The Service Management Board (SMB), which replaced the Service and Information Management Board (SIMB), has provided leadership and direction throughout the six years of the Initiative. The Board has proposed strategies for implementing recommendations from the Operational Review of the Service Delivery Infrastructure; guided the development of service policies and legislation; and, administered the implementation of the Government of Canada Service Vision. The Board has recommended resources, and provided direction and oversight to GOL operations including a close-out strategy and transition to service transformation.

In developing the Logic Model for the GOL Initiative, it was useful to begin by identifying and understanding its objective(s). It is has been observed that the GOL Initiative has presented a collectivity of several overlapping objectives and themes that have evolved and have been articulated slightly differently over time. As the GOL Initiative progressed, its opportunities and potential impact on Canadians and public servants, and on the relationship between Canadians and their government became clearer. The focus or emphasis of the GOL Initiative objectives was refined as the GOL Initiative evolved.

During the interviews conducted to develop this Framework, senior level stakeholders expressed many views regarding the objectives of GOL and the impact that GOL has had and continues to have on areas of activity well beyond those of information management (IM), information technology (IT) and service transformation.

The strategy to data collection taken in this Evaluation Framework was to utilize and build upon the good work already undertaken within and around the many facets of the GOL Initiative. For example, considerable measurement has been conducted and reported on the 130 on-line Government of Canada (GoC) most commonly used services regarding the progress made toward improved delivery of these transactional and information services, as well as toward evolution of enabling technologies and processes attributable to GOL.

The evaluation options presented in the Framework were in conformity with the requirements for rigorous methodologies for program evaluation as per the TBS Policy on Program Evaluation. The following three options were developed:

• Option A presented the minimum requirements for a retrospective evaluation of GOL accomplishments.
• Option B presented a more comprehensive approach that ensures that findings are supported by multiple lines of evidence, and takes into consideration the context of the broader picture of the evolution of GOL toward service transformation and e-government transformation.
• Option C presented all of Option B plus an added in-depth component on return on investment (ROI) and public value, particularly in light of recent groundbreaking work done in this area in other jurisdictions and countries.

Nearing the completion of this Evaluation Framework, the Director General (DG), AEB, and CEO, ITSB, agreed that Service Management Board (SMB) was the authoritative entity in respect to investments in GOL including how to close out the initiative.

ITSB presented SMB with the evaluation options put forth in this Framework along with an option to report on the results of GOL using existing reporting instruments.

The alternate option for reporting GOL results presented to SMB, details of which are not included in this Evaluation Framework, is to take advantage of already existing processes to gather evidence, which would then be used to produce an enhanced Final GOL Annual Report to Parliament for March 31, 2006. SMB selected this latter non-evaluation option. An excerpt of the Record of Decision of the May 5th, 2005 SMB meeting is included in the conclusion of this report.

1.0 Introduction

In November 2004, the Audit and Ethics Committee (AEC) of the Department of Public Works and Government Services (PWGSC) approved a request from the Office of the Chief Executive Officer of Information Technology and Systems Branch (ITSB), PWGSC for "... help in the development of an Evaluation Framework for the Government On-Line Initiative, in fiscal year 2004/05, which would set the stage for future evaluation activities of GOL". The request was added to the 2004/05 PWGSC annual Evaluation Plan for the Audit and Ethics Branch (AEB).

1.1 Background and History

The Government of Canada has been working towards making its 130 most commonly used services available on-line by 2005. The commitment "to be known around the world as the government most connected to its citizens" was first articulated in the Speech from the Throne, October 12, 1999. This was followed by a further announcement in the December, 2001 budget indicating that "...the Government is providing $600 million over the next four years to implement the Government On-Line strategy...."

The Internet explosion had opened a new channel for delivering information and services to Canadians, Canadian businesses, and international audiences. The GOL Initiative had set out to lay the foundation pieces for the use of this channel as a vehicle to offer around the clock, convenient, efficient and cost-effective information and services that citizens and businesses expect from a modern government.

GOL was conceived to establish the foundation for e-government in all Canadian government organizations by targeting key technological, policy and service delivery. Advances could be rapidly shared and implemented by federal organizations and organizations of other government jurisdictions throughout Canada thus ensuring better value on technological investments.

"Through Canada's Government On-Line (GOL) initiative, the most commonly used federal information and transaction services will be available on-line by the end of 2005. The approach GOL has taken is citizen- and client-centred, as well as ‘whole of government’; the overall objective is to put federal services on-line in a manner that increases client satisfaction with services, while also improving the operational efficiency of service delivery." ¹

While this 2004 statement of achievement, approach and overall objective of the GOL is in keeping with early statements of intent of the GOL initiative, careful reading of GOL and GOL-related documents and consideration of stakeholder interviews suggests the complex evolution of this initiative.

For example, the bedrock of the GOL Initiative and the foundation upon which it has been built are found in the oft-quoted items from the Speeches from the Throne of 1999 and 2001:

Earlier statements suggest a somewhat broader approach was being considered. In October 1999, the government’s IM/IT strategy for the future was situated in the context of societal developments:

"The impact of technology is transforming our lives.

The Government of Canada feels this impact. As societies and economies evolve, largely due to the influence of globalization, changing demographics, and the rise of the digital economy, governments are "reinventing" themselves to meet new expectations and the priorities of citizens and businesses. These dynamics are compelling the federal government to create a new vision for its relationship with Canadians. Characterized as citizen-centred government, it is a vision that recognizes the different ways that people interact with their government:

• as taxpayers who expect value and results;
• as clients who expect accessible, quality services; and
• as citizens who participate in the democratic process." ⁴

Earlier in that summer of 1999, a technologically enabled service vision of government on-line was also enunciated.

"By pushing forward aggressively to get government on-line, the Government of Canada is playing an important role in making our country one of the most connected nations in the world. Government is becoming a model user of information technology, drawing on its considerable IM/IT infrastructure to bring integrated, accessible service to Canadians. As we do this, we are improving the quality of the information we provide, improving service delivery, and ultimately, becoming a world-class, technology-enabled organisation." ⁵

It is evident that the GOL Initiative was established without excessive formal parameters, beginning as a broad, all-encompassing initiative that was intended to evolve. Quite quickly it evolved into something more practical, as precise ideas and terminology emerged (e.g. gateways and clusters, secure channel), transforming concepts and notions into practice. The chart in Appendix A provides a sense of the pattern of this evolution. As the analysis of government services took place, the 130 of the "most commonly used" ones were selected for on-line development Much of the efforts and budgets of the "middle years" of GOL - about 2001 through much of 2004 - were focused on the monitoring the progression of these services toward the e-channel delivery and the funding of projects to support the transformation of some of these services.

While this was happening, the GOL governance committees began to appreciate the implications and potential of bringing information together that was enabled by putting services on-line. When the Canada Site architecture was developed, it was recognized in a practical way that on-line information grouping, clustering, and integration presented powerful possibilities. The "sleeper trigger" to the above (as was expressed by one interviewee) was the development of Canada’s "Common Look and Feel". Government now appeared to be one, single, integrated enterprise organized to serve citizens. Within the appearance of coordinated information in gateways and clusters, one could see patterns and envision seamless service delivery - service transformation.

While for some, GOL brings to mind secure channel and/or e-government, increasingly considerations of "people change" in the Pubic Service also came to the fore. Often the "people" discussion took place before "technical" or "service" issues were considered. Recently this "softer" people side of things has also been reflected in GOL statements, drawing attention to and supporting the cultural changes needed to enable and sustain the evolution of horizontal, collaborative, multi-jurisdictional service delivery environments.

In retrospect, the evolution of GOL can be likened to an hourglass, with very broad all encompassing objectives expressed in the early days, followed by fairly specific objectives related to putting services on-line, and then followed by very broad service transformation objectives again. The GOL Initiative, a catalytic and practical expression of the client-centric and whole-of-government concepts, has enabled these concepts. From this flows transformation.

The work under GOL has been organized into a number of components that operated under Treasury Board Secretariat (TBS) until December 2003. These include:

• the grouping of government information and services for easier on-line access by clients (Gateways and Clusters);
• the transitioning to on-line service delivery of government programs delivering most commonly used services (On-line Services);
• the development of a secure infrastructure over which transactions can be completed (Common Infrastructure/ Secure Channel);
• special projects and work to ensure organizational readiness (Organizational Readiness Office) of government to work more "horizontally", i.e., between and among government departments and agencies; and,
• the development of strategies and policies to direct and coordinate the federal presence on-line (Policy).

1.2 GOL Governance Structure

The initiative found its first home within the Chief Information Officer Branch (CIOB) of the Treasury Board Secretariat (TBS) as the GOL Initiative. On December 12, 2003, the responsibility for GOL was transferred to the Information Technology Services Branch (ITSB) at PWGSC. The functions transferred included:

• meeting the GOL objective of offering the most commonly used services on-line by 2005 and the development of gateways and clusters based on the needs and preferences of Canadians;
• the continued development and operation of the Secure Channel; and,
• the development of a change management and human resource framework to prepare the Public Service to work in a multi-channel service delivery environment.

The following key, senior level committees have been active in the governance and direction of the GOL Initiative:

1) TBSAC Information Management Sub-Committee of Deputy Ministers (TIMS)

TIMS’ membership includes a number of Deputy Ministers (heads of government departments). Its mandate is to provide advice for Ministers and leadership across government as regards initiatives to improve services to the public, including Government On-Line (GOL), and the management of information holdings and related technologies and of infrastructures within government. TIMS is responsible for overseeing the Strategic IM/IT Infrastructure Initiative (SII), aimed at providing a common secure infrastructure for electronic transactions, and Government On-Line (GOL), with its emphasis on single-window service delivery and ease of access through the transformation and integration of services for electronic delivery to the public.

2) Service Management Board (SMB)

The Service Management Board (SMB), which replaced the Service and Information Management Board (SIMB), oversees the transformation and delivery of multi-channel services to external clients, citizens and stakeholders. SMB proposes strategies for implementing recommendations from the Operational Review of Service Delivery Infrastructure, guides the development of service policies and legislation, and administers the implementation of the Government of Canada Service Vision. The Board also provides oversight to conclude GOL operations including a close-out strategy and transition to service transformation.

3) GOL Leads

The GOL Leads Forum is comprised of senior managers at the Assistant Deputy Minister and Director General levels responsible for coordinating GOL within their departments and agencies. Membership is not restricted to core departments. The chair of the GOL Leads Forum rotates with the host department. Meetings are held five times per year. The TBS Chief Information Officer chairs the annual stocktaking meeting extended to the chairs of the Service Transformation Committee (STC), the Architecture Review Board (ARB), the Information Management and Policies Committee (IM&PC) and Community Champion committees (IT Champions Committee, IM Champions Committee and Service Delivery Champions Committee).

1.3 Purpose and Scope of the GOL Evaluation Framework

The Government of Canada Evaluation Policy revised on April 1, 2001, states that "...public service managers are expected to define anticipated results, continually focus attention towards results achievement, measure performance regularly and objectively, and learn and adjust to improve efficiency and effectiveness. Managers must be accountable for their performance to higher management, to ministers, to Parliament and to Canadians".

This Framework has been developed as a tool for GOL managers in departments and agencies, and at TBS and PWGSC, to help in measuring and reporting on results being achieved by the GOL Initiative. It is intended to serve as a comprehensive and reliable instrument for the evaluation and reporting on the performance of this major technical, culture change and learning initiative of the Government of Canada.

1.4 Framework Development Methodology

A GOL Evaluation Working Group comprised of representatives from each GOL component was established for this project, and has worked closely with the PWGSC project authority and CAC contract resources in the development of this Framework. The specific methodologies and activities undertaken for this project include:

• Review of documentation and existing reports (see Appendix D);
• Identification of stakeholders and development of comprehensive interview and guides;
• Interviews with relevant stakeholders to validate an understanding of the issues, challenges, evolution and approaches taken to the GOL over its lifespan (see Appendix E);
• Development of a comprehensive Logic Model identifying inputs, activities, outputs, and outcomes (immediate, medium, and long-term) for each GOL component;
• Consultation with the Working Group on the Logic Model and evaluation issues;
• Identification of the issues and associated questions to be addressed through the evaluation exercise concerning relevance, success and cost-effectiveness for each component;
• Prioritization of the evaluation issues and questions based on formal Government of Canada reporting requirements and feedback from stakeholders;
• Identification of data requirements the development of a data collection strategy outlining key data requirements, data sources and associated options;
• Development of an evaluation matrix documenting the evaluation issues, questions, indicators, methodologies and data sources;
• Development of alternative strategies for proceeding with an evaluation – describing for each the data collection approaches/methods and data sources, costs and other resource requirements, limitations, and outputs;
• Drafting of a comprehensive report and presentation material; and,
• Finalization of the Evaluation Framework report.

2.0 Component Profile of GOL Initiative

Since its inception, the GOL Initiative has been a continuous work in progress. As indicated in the Background and History section of this Framework, GOL was conceived to establish the foundation for e-government in government organizations by targeting key technological, policy and service delivery advances that could be rapidly shared and implemented by government organizations at all levels throughout Canada.

Today, many discussions and presentations on the subject of technology and service delivery no longer mention GOL specifically. Rather, they deal passionately with issues of e-government, e-business, e-service, and even e-society and the culture change that will accompany it – all ideas that were nascent in GOL, and that continue to occupy centre stage, even as the period of centralized GOL leadership and funding comes to a close.

In this section, the key components of the GOL Initiative are briefly described, and the Logic Model is presented and discussed along with the GOL outputs and outcomes.

2.1 Description of Components

The five components of the GOL Initiative are: Policy; Gateways and Clusters; On-line Services; Secure Channel; and Organizational Readiness Office.

Component 1 – Policy and Standards

The Policy component of the GOL Initiative involves the development of policies and standards to direct and coordinate the federal presence on-line, including the Common Look and Feel Standards, and the IT security standards and privacy standards. As well, this component advises on the legislative impact of the GOL Initiative as well as supporting the governance of the initiative.

Component 2 - Gateways and Clusters

Gateways and Clusters represent a key element in the delivery of client-centric on-line information and services. By providing Web-based groupings of service offerings designed from the perspective of the needs of identified client groups, Gateways and Clusters continue to integrate information about and access to services that were previously scattered across various departmental Web sites. The Gateways and Clusters component was developed as a means to organize departmental information and services available on the Canada Site on a subject and client-centric basis. Public opinion research and client feedback revealed that many Canadians searching for government information and services on the Canada Site were often confused by the departmental organization of services. To render on-line services more relevant and accessible to clients, the Cluster Blueprint was implemented in 2000 as the framework for the reorganization of the Canada Site into a set of three access "gateways" – Canadians, Non-Canadians and Business – that represent the three main client groupings or segments of the government. Within these major groups, numerous portals ("clusters") reorganize and position information and services found on departmental Web sites to provide more effective access to government by subject (e.g., health) or by audience (e.g., seniors). The role of the Gateways and Clusters is to act as the catalyst for the integration of information and services across federal departments and agencies by bringing together partners with related interests serving common clients.

Gateways and Clusters management and governance is provided through the interdepartmental Gateways Steering Committee. Some $95 million of GOL funding has been allocated to Gateways and Clusters over the period 2000-01 to 2005-06 to support implementation and development. Gateways and Clusters activities can be grouped into four key areas:

• Leadership and Direction: development and communication of clustering priorities and principles;
• Community Building and Partnerships: nurturing valued horizontal and vertical collaborative relationships among cluster partners;
• Clustering: site development and implementation of common information/ content management guidelines based on general preferences of client groups; and
• Client Insight: public opinion research and focused surveys in order to understand client priorities and needs.

By creating an environment in which partners can collaborate across government jurisdictions to deliver integrated services, the Gateways and Clusters component is intended to play a key role in the improvement across service delivery channels. Intended benefits or outcomes, including easier and more convenient access to services contribute to the increased relevance and satisfaction with government on-line services and an increased take-up of online service delivery as the channel of preference.

Component 3 - On-Line Services

The On-Line Service Delivery has two folds: the progression of the 130 Government of Canada (GoC) most commonly used services toward the e-channel based on defined target and the funding of various initiatives to support the transformation or improvement of some of these services.

Early on in the history of the GOL Initiative, analyses of government services were conducted in departments and agencies. This work resulted in the identification of the 130 services that touch the lives of the greatest number of Canadians or, the most commonly used services, both informational (48%) and transactional (52%). For each of them, an on-line progression target was established to improve access and increase the on-line availability of credible and relevant federal information, and to offer users a critical mass of transactional on-line services.

It was expected that by making available this critical mass of appropriately marketed on-line services, and by ensuring that all individuals and businesses desirous of using the Internet channel could do so, an increase in client take-up would result. This increase would occur both in the frequency of usage and in the adoption of new transactions offered, as clients became aware of the benefits of using on-line services, including more complete and accurate information access, time savings, cost savings, and simplified interactions. The evolution toward the e-channel has resulted in extensive information holdings increasingly being made available through the Internet, providing either information or transactions. As this happens, departments and agencies are upgrading processes to manage these holdings, to ensure that they are accurate and up-to-date, and to offer them in plain language formats.

The 130 GoC most commonly used services are distributed among 34 federal departments and agencies participating in the GOL Initiative. The following table provides a sense as to the proportion of services being accessed by Client User Groups.

Exhibit 1 Commonly Used Federal Information and Transaction Services Proportion Accessed by User Group
		Proportion of Services
Client "User" Group	Number of Services	Total	Transactional	Informational
Canadians and Canadian Residents	85	65%	33%	66%
Canadian Businesses	35	27%	80%	20%
Non-Canadians and Government	10	8%	66%	33%

To support the implementation of the e-channel, a series of business cases, pilots and projects were funded and closely monitored. These initiatives involving some of the 130 most commonly used services promoted horizontal and "joined-up services", as well as the transition to the e-channel and the use of the common secure infrastructure, where applicable. A few of the smaller agencies were funded for one or two projects, many departments received funds for 4-8 projects, and larger departments could have been funded for as many as 17 projects.

The availability of informational and transactional services has had considerable impact on usage and interaction patterns. In 2002, slightly over 20% of completed client-government transactions were done on-line. This rose to 30% in 2004. In 2001 slightly over 150 million interactions with government took place on-line. This increased to some 600 million in 2004. Approximately half of the GoC most commonly used services involve other jurisdictions or the private sector.

Component 4 - Secure Channel

The Secure Channel component is developing a common high-speed infrastructure to enable Canadians to conduct secure and private electronic transactions with federal departments and agencies and other participating jurisdictions through its secure and reliable network. It offers services for security, registration and authentication that enable departments and agencies to deliver the most commonly used services on-line. With the common infrastructure and services supplied by Secure Channel and the assurances that it provides with respect to security and privacy, the GOL Initiative’s vision of client-centric, trustworthy and accessible, cross-government, multi-jurisdictional service anytime, anywhere can be realized.

The Secure Channel component develops and provides additional products and services for security, registration and authentication to departments and agencies and other participating jurisdictions. In addition, it manages the technical components of the infrastructure. The vendor selected to build the infrastructure is a consortium of global technology companies selected in 2001 and led by Bell Canada Enterprises (BCE).

In particular, Secure Channel is designed to meet the following service objectives:

• Broad Accessibility to Government Programs - government information and services are accessible regardless of location, time or demographic group;
• Government as an Integrated Enterprise - a seamless view for all GoC information and services both internally and to the Canadian public;
• Privacy, Security and Authentication Efficient - the information and services are authenticated and protected to the required level; and
• Scalable Service Delivery - able to respond quickly to changing service expectations.

Since its activation in 2002, the Secure Channel has grown to serve over 30 active or pending applications in 22 federal departments with more expected. The services are oriented along two lines, namely:

1. Client-facing external services enable federal departments and agencies to provide on-line services to Canadians; and
2. Government-facing internal services offer specialized services for GoC public servants and trusted partners.

All federal departments and agencies now use Secure Channel's network infrastructure and operations services (SCNet) to connect to the Internet and to enable citizens’ access to their services. SCNet replaced the previous Government Enterprise Network (GENet) in the fall of 2003. Several services for Canadians, businesses and the GoC are now available:

• CRA’s Address Change On-Line (launched September 2002) allows taxpayers to verify or change their home and mailing addresses and telephone number on-line;
• HRSDC’s Web Record of Employment (ROE) (launched May 2003) enables Canadian businesses to file ROE forms on-line. To date, 1,219 employers are using HRSDC’s Web ROE. About 96,000 transactions have been processed to date;
• Rollout of Compensation Web Applications (CWA) on-line services for GoC employees in spring 2004. Web-accessible applications include pay-stub details and personal pension information; and
• Receiver General’s on-line payment service uses the Secure Channel to enable Canadians and businesses to pay for GoC goods and services on-line. The service, known as the Buy Button, has processed about 108,000 on-line payments for seven departments’ storefronts.
• Common Registration Service (CRS) enables Canadians to use a single set of credentials for authentication and registration to access several departmental applications. The Canadian Radio-Television and Telecommunications Commission (CRTC) was the first department to use CRS with its telecom and licensing service (March 2004). Other departments and agencies integrating their on-line services with CRS include Passport Office, Immigration and Refugee Board, and Health Canada.

Secure Channel will help support the Statistics Canada 2006 Census by assuring the trusted delivery and return of an on-line questionnaire (to fill out and submit) to citizens in 13.5 million households.

Component 5 - Organizational Readiness Office (ORO)

ORO identifies and helps departments address culture and workforce issues related to the implementation of GOL and Service Improvement initiatives. It is a strategic unit that leads horizontal initiatives to improve capacity, agility, and the management of human capital in the public service in anticipation of the transformation of federal government business processes, working through the Information Technology (IT), Information Management (IM) and Service Delivery (SD) communities. These initiatives are critical to the success of GOL, Shared Services, Management of Government Information (MGI), and other policy and change agendas. Working with departments, agencies and these three communities, ORO has developed and is beginning to implement competency-based Human Resource (HR) Frameworks made up of proven practices and approaches.

Given its unique position at the intersection of interests (i.e., communities) and authorities (i.e., departments and agencies), the ORO is positioned to address strategic rather than transactional HR issues. For example, the ORO focuses on HR initiatives that are a catalyst for development of horizontal relationships across the federal government and that build both departmental and community capacity to coordinate and share cost efficient and effective work practices.

ORO engages senior leadership through committees of Community Champions at the Director General and Assistant Deputy Minister level. The IT and IM Champions Committees are sub-committees of the Chief Information Officer Council (CIOC). The CIOC and the Service Delivery Champions Committee reports to the SMB, thus, both CIOC and SMB ultimately report to TIMS. In addition to the influence they have on whole-of-government decisions at the CIOC, SMB and TIMS levels, the Champions individually play key roles in leading department-specific change-management initiatives within their respective organizations. The understanding that they develop of broader "enterprise" and community objectives helps ensure the decisions that they make or influence are well aligned with government goals.

ORO focuses on:

• Awareness and Engagement activities that communicate the "whole of government" perspective allow better, more informed decision making by public servants both in their formal organizational roles and in making personal career and development decisions.
• Building Human Capacity and Agility to respond to changing technology, legislation, government objectives and public expectations and needs, while continuing to deliver the business of government in ways consistent with Canadian and public service values.
• Improving the flow of shared work practices (including tools and applications) so as to accelerate adoption of proven practices and reduce the learning curve for managers and employees.

2.2 Stakeholders

There are five groupings of GOL Initiative stakeholders:

1. Users of Government Information and Services;
2. Federal Government Programs;
3. Public Service Members;
4. Other Canadian Government Jurisdictions; and,
5. IT Consortium.

The first group of stakeholders is the clients or users of government information and services ⁶, specifically Canadians (or citizens) and Residents, Non-Canadians (individuals, businesses and institutions), and Canadian Business. The GOL Initiative was created to make it easier for Canadians and other clients to access government services by providing an additional channel and to renew the government’s relationship and trust with its citizens. Users are expected to benefit, in terms of saving time and money, from the GOL Initiative by having quicker access to government information and transactions, and having electronic access to so much more information. Through the improvement of services to Canadians and Canadian Businesses, an increase to commercial opportunities abroad and to Canada's visibility was expected, thus strengthening the Canadian international presence.

The second group of stakeholders is the federal government programs and their 34 hosting departments and agencies ⁷ that are responsible for the 130 GOL services, i.e., the providers of information and services

The third group of stakeholders is the members of the Public Service from Deputies to front-line service delivery workers whose jobs and accountabilities will be affected by the changes and transformations being brought about by and resulting from the GOL Initiative. More specifically, this group of stakeholders includes the Public Service communities of interest (i.e., IT, IM and SD), who support the OGDs in providing information and services.

The fourth group of stakeholders includes other jurisdictions in Canada that (want to) deliver services on-line and are willing to partner with the federal government.

Finally, the fifth group comprises the consortium of global technology companies involved in developing the enabling technologies, including the Secure Channel, tools to deliver services on-line, and related products and services. They will benefit directly from their contracts with the government and indirectly by becoming, if they are not already, experts in the area of e-government and marketing this expertise and enhanced knowledge to other jurisdictions in Canada and abroad and to the private and non-profit sectors. This stakeholder group would also include other Canadian companies that can benefit from applying the technologies and tools.

2.3 Logic Model

A Logic Model is an essential tool in conducting an evaluation study. It is intended to show, in a succinct fashion, the logic of how a program, policy or initiative is expected to achieve its objectives. Generally, a Logic Model builds on the management process and graphically displays the linkage and causal connections between the program’s (or policy’s or initiative’s) activities that are supported by inputs and the different levels of results – including outputs, strategic outcomes (immediate and intermediate) and ultimate results (or long-term outcomes).

It is useful to examine this process as a sequence of cause and effect relationships. For example, the inputs make it possible for the activities to be carried out; the activities then produce the outputs; the outputs, in the hands of the recipients, lead or are expected to lead to achieving immediate and intermediate outcomes, which generally correspond to desired changes in terms of knowledge, attitudes, skills and behaviours of recipients and any subsequent targeted people or organizations.

Generally, intermediate and ultimate outcomes clarify implicitly or explicitly who are the intended beneficiaries and targets of the program, policy or initiative. They can include specific groups of Canadians, Canadian or foreign businesses and non-profit organizations, a specific department(s) or jurisdiction(s), or the people of Canada in general. In turn, the intermediate and ultimate outcomes are intended to lead to achieving the ultimate outcomes or results, i.e., the expected long-term changes.

Developing a Logic Model facilitates the examination of the plausibility of the various relationships and provides an invaluable tool for the on-going and future performance monitoring and evaluation of the program, policy or initiative.

The key elements for developing a Logic Model for the GOL Initiative include the following:

1. Activities: What key activities are being undertaken to contribute to the strategic GOL outcomes? What work is carried out to produce the outputs?
2. Outputs: What outputs need to be produced to achieve the expected strategic GOL outcomes (e.g., policies and standards, the actual gateways and portals, justification and marketing information, a Secure Channel, HR Frameworks, and tools)?
3. Immediate outcomes: What short-term consequences should be expected to result from the key GOL activities and outputs? How do recipients of outputs use the outputs to effect changes in their knowledge (including awareness), attitudes, skills, intentions, and behaviours of the target beneficiaries?
4. Intermediate outcomes: What strategic GOL outcomes are expected to be achieved (flowing from activities, outputs and immediate outcomes) that will demonstrate progress towards achieving the long-term outcomes and ultimate results? Are the targeted stakeholders being affected in the way that they were expected to be?
5. Long-term outcomes/Ultimate results: Why has the GOL Initiative been undertaken and what is it expected to ultimately achieve and who are the ultimate beneficiaries? Long-term outcomes (and sometimes intermediate outcomes, as well) may take a very long time to achieve and are generally subject to influences (both positive and negative) beyond the GOL Initiative.

In developing the Logic Model for the GOL Initiative, it is useful to begin by identifying and understanding its objective(s). It has been observed that the GOL Initiative has presented a collectivity of several overlapping objectives and themes that have evolved and have been articulated slightly differently over time. As the GOL Initiative progressed, its opportunities and potential impact on Canadians and public servants, and on the relationship between Canadians and their government became clearer. As the GOL Initiative evolved, the focus or emphasis of the GOL Initiative objectives were refined.

Logic Model of the GOL Initiative - Draft (View detailed figure by clicking on link)

2.4 Activities to Outcomes

2.4.1 Activities and Outputs

The Logic Model depicts how GOL’s five components contribute with the production of outputs and the achievement of outcomes in support of objectives. The components’ activities are:

Policy
These activities centre on the development and communication of policies, strategies, and standards, monitoring and reporting. Policies, strategies, standards, frameworks and reports represent the key outputs of these activities.

Gateways and Clusters
Working with the community of departments and agencies that are using the on-line channel, Gateways and Clusters provides leadership and direction developing and communicating clustering priorities and principles, establishing and nurturing horizontal and vertical collaborative partnerships, developing sites and implementing common approaches to content management, and conducting surveys and research into client priorities and needs. Key outputs include integrated portals, common tools for content management, collaborative arrangements and partnerships and the "Cluster Blueprint".

On-Line Services
The activities of On-line Services include the analysis of exiting and potential on-line services to identify opportunities for further development enabling most beneficial access, considering cost savings and cost avoidance implications of alternative initiatives, promoting increased usage of the channel as a service delivery mechanism, funding business cases, conducting pilots and projects, catalytic and pathfinder initiatives. Outputs also include on-line access options for informational and transactional initiatives.

Secure Channel
Activities in this component are concerned with the development and management of the operations of a secure common, high-speed, hardware and software infrastructure over which Canadians can conduct secure and private transactions with the federal government and other participating government jurisdictions. The contractual relationship between the government and the private sector consortium engaged to build and operate the Secure Channel is managed through this activity stream. Outputs include an available, secure, reliable, trustworthy, high-speed network for Canadians, as well as security, registration and authentication products for government.

Organizational Readiness Office
Organizational readiness provides a human resource and cultural focus to the workforce change and transformation aspects of the implementation of GOL and Service Improvement Initiatives. Activities include providing leadership to horizontal initiatives to improve capacity, agility and management of Information Management, Information Technology and Service Delivery communities. Outputs include competency based frameworks, learning and knowledge sharing networks, and community development and leadership tools.

2.4.2 Outcomes

Immediate Outcomes
As legislation and policies are modified to become Internet friendly, and federal government services are migrated to the new channels, users are provided with an additional means to accesses government information and services. Easier and simplified access is supported with the enhanced privacy and security of electronic transactions. Satisfaction is expected to increase as consistency and accuracy of information provided is facilitated across channels. Knowledge of government services and activities is expected to increase, thus, increasing transparency and accountability. Savings accrue to users in terms of cost avoidance and cost savings. Internally within government, process redundancies and costs are reduced as services are integrated and made available as and when requested. The capacity of the public sector to support integrated and online service offerings is increased.

Intermediate Outcomes
Increased GoC presence on the Internet is expected to lead to increased take-up of the on-line channel. Government will be brought closer to citizens reducing time, effort and costs as they access information and services in a manner most convenient to them. Increased access will result in increased knowledge of government services and activities which will promote an increase in the quality of the information and services provided. Knowledge and involvement by Canadians in government activities will promote increased trust and confidence in the information and services, and On-line Service delivery. Government’s On-line Service delivery capacity will enhance the capacity of private sector partners to provide service delivery mechanisms to other jurisdictions and the Canadian private sector’s use of on-line business tools. Increased transparency and knowledge of government activities and policies will promote increased and more knowledgeable compliance. Knowledgeable Canadians will drive an increase in the quality of services which will be provided by an increasingly agile public sector that has the capacity to adapt and seize the opportunities presented by Service Improvement Initiative.

Long-Term Outcomes
Canadians will receive better and more responsive government services as they become more engaged, accessing information and services on-line. Government will be closer to citizens. Operational efficiencies within government will increase as service transformation increases agility, and delivery is increasingly citizen centric. Multi-jurisdictional service transformation will increase, promoting increased productivity and economic growth as savings accrue directly and indirectly to citizens and businesses. Governments will serve an increasingly knowledgeable and engaged Canadian, experienced in the use of the on-line channel. Canada’s reputation as a government connected to its citizens - a country having a modern government leading a knowledge-based economy and society will be enhanced.

The development of the Logic Model for the GOL Initiative began with consideration of objectives. There were many objectives that evolved over the years. Underlying these objectives was the notion that the opportunity presented by the Internet and its reach might be harnessed through the application of technology and innovative thinking to foster knowledge creation and sharing, creating a base to enable and nurture a whole of government, citizen focused transformation of government service delivery.

2.5 GOL Funding

A table presenting the approved allocation of central GOL funds by department/agency is included as Appendix C of this report ⁸.

3.0 Understanding the GOL Context

Senior level stakeholders have expressed multiple perspectives on the objectives of GOL and on the impact that GOL has had and continues to have on areas of activity well beyond those of IM, IT and service transformation. Indeed, the notion of transformation pervades all thinking about GOL.

The transformation is not solely about service transformation. Rather, it is about fundamental Public Service transformation. The changing GOL technology of the last five years has enabled profound service transformations that now necessitate the transformations of organizations, service delivery, human resources, and governance – and all of this continues to be a work in progress. None of it is complete, and no one in the GOL community believes that it can ever be complete. Through GOL and into the new environment of decentralized, multi-departmental and multi-jurisdictional one-window/multi-channel service delivery, Canada has entered the real world of continuous change, and its Public Service institutions and constructs – including notions on how to conduct program evaluations – have to adapt and be agile to remain relevant.

It is in this context and at this point in the evolution of GOL that the development of this Evaluation Framework has taken place. It is a Framework that strives to respect both the nature of the change and transformation that is being experienced and the requirements for rigorous evaluation methodologies that can reliably assess outcomes and value through the application of measured, multiple lines of inquiry, applied with due consideration to economy and efficiency.

In its Final Report of December 2003 to the President of the Treasury Board entitled Connecting with Canadians: Pursuing Service Transformation,⁹ the GOL Advisory Panel comprised of representatives from the general public, and the business, high-tech, academic, education and voluntary sectors reviewed some of its own earlier comments on GOL, and it is instructive to note how broad its vision of GOL was from early on. The report states, referring to the period before 2002 that:

"We quickly came to the conclusion that the GOL initiative is about much more than putting federal government services on-line. As we stated in our First Report of April 2002, '...we have been struck by the sweeping potential of this initiative both for the direct changes it is starting to bring and as a catalyst for far broader indirect change. Because this initiative, in its essence, involves changing the ways Canadians communicate with each other and their government, it has the potential to change - and in largely unpredictable ways - the most basic relationships that underpin our society. [...] All of this implies that government must be not simply restructured but reconceived to accommodate, but just as important to anticipate, the waves of change that will erode traditional relationships even as new ones are created'."

It appears that the work to date on performance measurement, performance reporting, annual reporting, comparative international reporting, audit, and other evaluative exercises has addressed some of the change elements alluded to in the statement above.

The evaluation issues articulated in this Framework take into consideration the sweeping potential of direct and indirect changes being or expected to be brought about by the GOL Initiative. These changes are reflected as outcomes in the Logic Model. This approach considers the changes being brought about by GOL and the essential issues of: governance; sustainability of the GOL or its successor iteration, manifestation or expression as a horizontal initiative; agility of the Public Service to adapt to continuous change and continuous service transformation in the context of on-line service delivery; and approaches to cost-effectiveness in this environment. A future evaluation will be cognizant of the ever-present backdrop of security, privacy and legal considerations within which GOL operates.

A number of other observations have informed the development of this Evaluation Framework and will be given due consideration in an evaluation. Among these are the following:

Knowledge Based Society

• The overall development toward and the strengthening of the knowledge-based society in Canada has been noted by many in the IM/IT world over the last number of years, and well before the Canadian GOL Initiative was announced. From its inception, GOL was cognizant of the importance of this background and development, and the “three tier” approach of 1999 explicitly recognized seamless delivery of many kinds of services – including knowledge services – as part of tier three. Indeed, many departments, which saw themselves primarily as knowledge-based rather than transaction-based in their relationships with citizens, felt that their GOL future lay in the imminent integration among departments of previously isolated blocks of knowledge in new and exciting ways yet to be envisaged - all coordinated and delivered through new service vehicles, packages and channels, also yet to be foreseen. This Framework recognizes the importance of this vision, and the observation that knowledge formulation/delivery transformation has not become reality to the same extent as transaction service transformation. How and why this came about will be explored in a subsequent evaluation.

International and Multi Jurisdictional Impact

• GOL is a multi-stakeholder initiative not only of federal government departments and agencies, but also of organizations and governments well beyond, both in Canada and abroad. Canadian provinces, territories and municipalities, Canadian companies, organizations and citizens, foreign governments and interests with whom Canada interacts, and foreign organizations and citizens seeking information, all have a stake in GOL. This Evaluation Framework recognizes the difficulties of dealing with issues of perceived and real public value and value-for-money when determining evaluation criteria and indicators in this context. In this regard, the experience of other jurisdictions in Canada and elsewhere, including in the European Union, Australia and other places have been researched. The Framework takes the various approaches into consideration, particularly when addressing issues of cost-effectiveness and value.

GOL Evolution

• GOL has experienced an extraordinary conceptual development over five years that has seen the articulation of its objectives change, expand and transform. The evolution of the statements of objectives and how government and the Public Service, and their accountability tools and mechanisms, accommodate or try to accommodate this kind of evolution, is a subject of lessons learned of which this Framework is aware and would be fully explored in a subsequent evaluation.

By developing this Evaluation Framework in the ways described above, it is expected that an evaluation based on it will satisfy requirements for the evaluation of GOL relevance, success, and cost-effectiveness, while successfully articulating lessons learned for the future.

4.0 Evaluation Issues

The following evaluation matrix presents the proposed evaluation issues, performance indicators, data sources and data collection methodologies, organized by the TBS Evaluation Policy categories of Rationale/Relevance, Success/Impact and Cost Effectiveness/Value.

The 2004 report, Performance Measurement for the Government On-Line Initiative¹⁰ (PMAF) developed by TBS for the GOL Initiative includes eleven principles and dimensions against which progress towards expected outcomes would be measured. The report provides a reasonable methodology regime to assess GOL progress of its fundamentals, and it will serve as a basic approach to guide a GOL evaluation. However, the evaluation proposed in this Framework goes further – by exploring the measures put in place to ensure long-term sustainability of GOL principles. The evaluation issues and questions identified do address the eleven PMAF dimensions and indicators: Convenience, Accessibility, Credibility, Critical Mass of Services, Service Transformation, Client Take-up, Client Satisfaction, Security, Privacy, Efficiency, and Innovation. The table, Concordance Between PMAF Dimensions/Indicators and the Evaluation Issues, in Appendix B displays this relationship.

The strategy to data collection taken in this Evaluation Framework is to utilize and build upon the good work already undertaken within the many facets of the GOL Initiative. For example, considerable measurement has been conducted and reported on for the 130 on-line GOL projects regarding the progress made toward improved transactional delivery and toward evolution of enabling technology and process attributable to GOL. The Framework proposes to use these measures in a future evaluation, but not duplicate the underlying analyses that are carried out as part of the PMAF. Rather, the proposed evaluation goes beyond the PMAF measures by measuring and assessing the successes, near-successes, and successes to be accomplished, in the realm of cultural change and transformation that GOL has enabled and unleashed, and by attempting to measure cost-effectiveness and value.

5.0 Approach to Cost Effectiveness

The value of the GOL Initiative specifically, and e-government in general, is in the benefits it delivers to the public and the new avenues it opens to create value. But investing in GOL and e-government can be costly, and its value to the public that supports it must be demonstrated.

A comprehensive methodology for measuring cost-effectiveness and determining return on investment (ROI) in e-government has not been fully developed. Such a methodology is proposed in Appendix G. Its intent is to examine the benefits of e-government as a whole, and compare these benefits to costs specific to investment funding from the GOL Initiative.

The Evaluation Framework describes an approach to evaluating and measuring the cumulative ROI of the GOL Initiative over the 2000-2005 time period. The assumption is that the GOL Initiative provides a foundation, based on its projects and endeavours, that has driven, and will continue to drive, investment and uptake in e-government across the government as a whole, and that the impact of the GOL Initiative should be examined from the perspective of its impact on the broader federal e-government context and on the capacity of the government to achieve service transformation.

The belief is that investing in and building a GOL foundation enabled and supported the development of e-government as a whole over the past five years and into the future. The approach attributes - and therefore will include - many of the benefits of e-government to GOL foundation projects and endeavours (e.g., Proofs of Concept, Pilots, Leadership, Infrastructure, Regulations, Legislation, Governance, Shared Service Solutions, Common Look and Feel, Accessibility, Bilingual Documentation). Net benefits of the entire five-year GOL Initiative investment, plus e-government benefits in general, may outweigh the costs of GOL projects and endeavours. The intent of the approach is to quantify the benefits and value of all the GOL projects and endeavours, in addition to as many e-government benefits that can be identified.

The approach to conceptualizing ROI is to consider "value" in its broadest and most comprehensive form: the value of IT, the value of e-business, service transformation and online capability enhancements (people, systems, processes, capacity and sustainability). A number of options for ROI analysis are outlined in Section 6.0 of this report. Options A and B limit the focus to a consideration of cost avoidance. The approach would be to quantify the value of the alternative solutions not selected. For example, a possible alternative to creating one Secure Channel infrastructure for all of the Government of Canada could have been to build multiple infrastructures, one for each of the large departments with intensive transaction delivery business processes. Significant upfront and on-going costs were avoided by not choosing that alternative. The same cost avoidance approach would be applied to other GOL components, including On-Line Services (the alternative of not having a unified approach to measuring service maturity), Gateways and Clusters (the alternative of not having a gateway and cluster focus to citizen/business interactions), Organizational Readiness (the alternative of not having a whole-of-government horizontal approach to culture change, skills training and retaining/recruiting strategies), and Policy Analysis and Development (the alternative of not having a coordinating central agency perspective and/or leadership). Option C involves conducting a full ROI value assessment and analysis, including assessing both the direct and indirect benefits of GOL and the benefits of e-government, and focusing the cost analysis on the upfront and on-going costs specifically of the GOL Initiative as the initiative that was built to create the foundation for e-government. The approach to the benefit and value analysis in this option would be based on consideration of multi-stakeholder impacts/effects, including citizens, business, NGO’s, provinces/territories, international competitiveness, stakeholder groups, public service employees, managers and executives.

6.0 Evaluation Options

All evaluation options presented below are in conformity with the requirements for rigorous methodologies for program evaluation as per the TBS Policy on Program Evaluation. In developing the options, the project team took into consideration the concerns and interests of the major broad stakeholder groups, including GOL Leads and project and endeavour managers, senior department and agency managers involved in on-going service delivery improvement and transformation, and other government and non-government parties concerned with present and future e-service and e-government transformation, as expressed in the research and interviews conducted for this project. Each of the evaluation issues, and associated data sources presented in the Evaluation Matrix, were examined in the context of the requirement for evaluation information to be useful for decision-making purposes by managers. The following three options were developed:

• Option A presents the minimum requirements for a retrospective evaluation of GOL accomplishments. The audience would principally be managers closely associated with GOL projects and endeavours. Option A ensures that evidence-based and credible findings will be obtained and presented on each evaluation issue. However, its scope would be limited.
• Option B presents a more comprehensive approach that ensures that findings are supported by multiple lines of evidence, and takes into consideration the context of the broader picture of the evolution of GOL toward service transformation and e-government transformation. It represents an evaluation that rigorously examines and sums up GOL accomplishments, successes and setbacks from different stakeholder perspectives, presents "lessons learned", and lays the groundwork for the development of evaluation parameters for service and e-government transformation in its various guises in the future. This option assures that the full benefits of the GOL Initiative are identified, captured and measured, that no stakeholders are excluded, and that the results of the evaluation can reliably be used for decision-making purposes in a “whole-of-government” environment. It is an option that addresses the needs of GOL and e-service delivery personnel and managers, and the transformation thinkers of the future. The limitation of Option B is that it does not examine the ROI of the GOL Initiative nor the question of measurement of value, outside the parameters of the information and data presently collected and made available by departments and agencies to the evaluators.
• Option C presents all of Option B plus an added in-depth component on ROI and public value, particularly in light of recent ground-breaking work done in this area in other jurisdictions and by jurisdictional associations, including in the European Union and by the Organization for Economic Co-operation and Development (OECD) E-Government Network¹¹. Option C proposes the development and use of an ROI approach to GOL and service transformation that will be specifically developed for the GOL Initiative in the Canadian context, the basis of which is outlined in Appendix G of this Evaluation Framework.
• The options and their estimated costs are summarized in the table below.

See Appendix F for a breakdown of estimated costs by data source for each option.

7.0 Conclusion

As indicated in the Introduction section and section 1.2 Governance Structure, GOL was originally supported and managed through the Chief Information Officer Branch (CIOB) of the Treasury Board Secretariat (TBS). On December 12, 2003, the responsibility to provide this centralized support and coordination role to GOL participating departments and agencies and to the GOL governance structure was transferred to the Information Technology Services Branch (ITSB) at PWGSC. Those components such as the Secure Channel, which had evolved past the design and feasibility phase into an implementation phase were also assigned to ITSB.

The Service Management Board (SMB) has provided leadership and direction throughout the six years of the Initiative. The Board has proposed strategies for implementing recommendations from the Operational Review of the Service Delivery Infrastructure; guided the development of service policies and legislation; and, administered the implementation of the Government of Canada Service Vision. The Board has provided resources, direction and oversight to GOL operations including a close-out strategy and transition to service transformation.

Nearing the completion of this Framework, it was agreed with ITSB that SMB was the authoritative entity respecting investments in GOL including how to close out the initiative. The Evaluation Directorate provided the Government On-line Initiative Directorate in ITSB with descriptions and costs estimates for the three formal evaluation options presented in this Framework. Because this proposed evaluation would involve partnerships with other federal organisations, with other levels of government, and with the private sector, ITSB also chose to present the proposed evaluation options to SMB members along with an option of their own, which would be much less intrusive to both GOL participating and non-participating organizations. This option takes advantage of already existing processes to gather evidence, which will be used to produce the final GOL Annual Report to Parliament for March 31, 2006, in lieu of a formal evaluation.

A copy of the record of decision of SMB’s May 5, 2005 meeting regarding their direction concerning the "...optimal approach for assessing the GOL Initiative given scarce financial and human resources" follows:

8.0 Appendices

Appendix A: Mapping GOL Strategic Objectives Year-Over-Year

Appendix B: Concordance Between PMAF Dimensions/Indicators and the Evaluation Issues

	PMAF 11 Dimensions & Indicators 12
	Citizen / client- centred government			Better, more responsive service				Capacity for on- line delivery
	Conve- nience	Accessi- bility	Credi- bility	Critical Mass of Ser- vices	Service Transfor- mation	Client Take- Up	Client Satis- faction	Secu- rity	Priva- cy	Effi- ciency	Inno- vation
GOL Evaluation Issues
Relevance
1. The extent to which GOL "client centric" and "whole of government principles and consistent with ongoing GoC priorities
2. The ongoing need to build capacity within the GoC to achieve long term e-channel benefits
Success
3. The extent to which GOL has succeeded in making the federal government the most known around the world as the government most connected to its citizens
4. The extent to which GOL has succeeded in improving the federal government’s service delivery
5. The extent to which GOL has contributed to changing the culture of the Public Service in terms of horizontal, multi-jurisdictional and collaborative service delivery approaches
6. The extent to which individual GOL undertakings have been completed as intended, outcomes achieved, and the causes of variances identified and understood
7. The extent to which individual GOL undertakings have succeeded in laying a sustainable foundation within GoC for ongoing transformation of service delivery
8. The extent to which GOL assisted the private sector in enhancing its competitiveness
9. Unintended outcomes (positive or negative) that occurred as a result of GOL
10. Lessons learned from the GOL initiative that can be applied more generally to ongoing transformation
Cost-Effectiveness
11. The extent to which user benefits (Canadians, business and international users), and benefits to government organizations in terms of increased resource efficiencies can be attributed to the GOL Initiative.

Appendix C: Approved GOL funding

Approved Allocation of Central Funds by Department/Agency; GOL Component and Year (December 2003)
Source: GOL On-Line 2004, Annual Report, Public Works and Governmenet Services Canada, March 2004

A) Gateways and Clusters

	Department / Agency (n = 18)		GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	Agriculture & Agri-Food Canada	Gateway & Clusters			612		400								1,012
	Canadian Heritage	Gateway & Clusters			400		40		283						723
	Canadian Institutes of Health Research	Gateway & Clusters			132										132
	Citizenship & Immigration Canada	Gateway & Clusters			1,155		400		300						1,855
	Communication Canada	Gateway & Clusters					2,970		1,615						4,585
	Environment Canada	Gateway & Clusters			1,000		20								1,020
	Foreign Affaires & International Trade	Gateway & Clusters	402		2,932		2,370		1,929						7,633
	Health Canada	Gateway & Clusters			1,672		500								2,172
	Human Resources Development Canada	Gateway & Clusters	328		7,203		3,650		5,081						16,262
	Indian & Northern Affairs	Gateway & Clusters					300		348						648
	Industry Canada	Gateway & Clusters	782		2,007		2,800		2,854						8,443
	Justice Canada	Gateway & Clusters			660		65								725
	Natural Resources Canada	Gateway & Clusters							555						555
	Public Works & Government Services Canada	Gateway & Clusters	3,622		3,600		1,200		2,862						11,284
	Solicitor General	Gateway & Clusters			200		15		535						750
	Statistics Canada	Gateway & Clusters	55		554										609
	Treasury Board of Canada Secreteriat	Gateway & Clusters					870		800						1,670
	Veteran Affairs	Gateway & Clusters	152		1,818		800		583						3,353
	GOL Funds Allocated as of December 2003		5,341		23,945		16,400		17,745		0		0		63,431
	GOL Funds (unallocated as of Dec. 2003)	Gateway & Clusters									16,855		15,000		31,855
	Total GOL Funds - Gateways & Clusters		5,341		23,945		16,400		17,745		16,855		15,000		95,286

B) Secure Infrastructure

	Department / Agency (n = 3)		GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	DND - Communications Security Establishment	Secure Infrastructure	5,565		10,335		10,050		6,350						32,300
	Public Works & Govermnet Services Canada	Secure Infrastructure	15,641		64,570		93,830		117,050		8,000				299,091
	Treasury Board of Canada Secretariat	Secure Infrastructure	11,284		32,394		11,851		13,500		12,000				81,029
	GOL Funds Allocated as of December 2003		32,490		107,299		115,731		136,900		20,000		0		412,420
	GOL Funds (unallocated as of Dec 2003)	Secure Infrastructure									59,410				59,410
	Total GOL Funds - Secure Infrastructure		32,490		107,299		115,731		136,900		79,410		0		471,830

C) Policy & Standards

	Department / Agency (n = 1)		GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	Treasury Board of Canada Secretariat	Policies & Standards	5,063		3,934		13,000		13,000		11,500		4,500		50,997
	Total GOL Funds - Policy & Standards		5,063		3,934		13,000		13,000		11,500		4,500		50,997

D) Business Case Development

	Department / Agency (n = 6)		GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	Canada Customs & Revenue Agency	Business Case					200								200
	Citizenship & Immigration Canada	Business Case					230								230
	Indian & Northern Affairs	Business Case					200		125						325
	Industry Canada	Business Case					200								200
	Natural Resources Canada	Business Case					200								200
	Veteran Affairs	Business Case					400								400
	GOL Funds Allocated as of December 2003		0		0		1,430		125		0		0		1,555
	GOL Funds (unallocated as of Dec 2003)	Business Case							0		0		0		0
	Total GOL Funds - Business Case		0		0		1,430		125		0		0		1,555

D) On-Line Services

	Department / Agency (n = 24)		GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	Agriculture & Agri-Food Canada	On-Line Services			4,327										4,327
	Canada Customs & Revenue Agency	On-Line Services	3,345		15,330		14,829		14,802						48,306
	Canadian Economic Development Agency (Quebec)	On-Line Services			25										25
	Canadian Heritage	On-Line Services	90		2,081		200		400						2,771
	Canadian Institutes of Health Research	On-Line Services			220										220
	Citizenship & Immigration Canada	On-Line Services	161		278		1,100		500						2,039
	Communication Canada	On-Line Services					1,000		1,300		100				2,400
	Federal Court of Canada	On-Line Services	50		150										200
	Foreign Affaires & International Trade	On-Line Services	29		2,603		900		2,224		5,528		1,917		13,201
	Health Canada	On-Line Services	790		5,829		3,115		3,970		5,515				19,219
	Human Resources Development Canada	On-Line Services	1,551		24,991		15,000		15,500						57,042
	Indian & Northern Affairs	On-Line Services			3,300		1,763		1,500						6,563
	Industry Canada	On-Line Services	625		5,879				3,156		875				10,535
	Justice Canada	On-Line Services			700		2,571		2,997						6,268
	Natural Resources Canada	On-Line Services			300										300
	Parks Canada	On-Line Services			850										850
	Public Works & Government Services Canada	On-Line Services	1,806		16,436		2,977		4,898		500				26,617
	Royal Canadian Mounted Police	On-Line Services	225		338										563
	Solicitor General	On-Line Services			301										301
	Statistics Canada	On-Line Services			2,603		1,600		1,800		1,000		500		7,503
	Supreme Court of Canada	On-Line Services			375										375
	Transport Canada	On-Line Services			300										300
	Treasury Board of Canada Secretariat	On-Line Services							2,953		1,985				4,938
	Veteran Affairs	On-Line Services			2,000		1,750		1,900		1,750		1,750		9,150
	GOL Funds Allocated as of December 2003		8,672		89,216		46,805		57,900		17,253		4,167		224,013
	GOL Funds (unallocated as of Dec. 2003)	On-Line Services									23,568		12,750		36,318
	Total GOL Funds - On-Line Services		8,672		89,216		46,805		57,900		40,821		16,917		260,331

D) Funding Summary

			GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	GOL Funds Allocated as of December 2003	Gateway & Clusters	5,341		23,945		16,400		17,745						63,431
	GOL Funds Allocated as of December 2003	On-Line Services	8,672		89,216		46,805		57,900		17,253		4,167		224,013
	GOL Funds Allocated as of December 2003	Secure Infrastructure	32,490		107,299		115,731		136,900		20,000				412,420
	GOL Funds Allocated as of December 2003	Policy & Standards	5,063		3,934		13,000		13,000		11,500		4,500		50,997
	GOL Funds Allocated as of December 2003	Business Case					1,430		125						1,555
	GOL Funds Allocated as of December 2003		51,566		224,394		193,366		225,670		48,753		8,667		752,416
	GOL Funds (unallocated as of December 2003)	Gateway & Clusters									16,855		15,000		31,855
	GOL Funds (unallocated as of December 2003)	On-LIne Services									23,568		12,750		36,318
	GOL Funds (unallocated as of December 2003)	Secure Infrastructure									59,410				59,410
	GOL Funds (unallocated as of December 2003)								0		99,833		27,750		127,583
	Total GOL Funds		51,566		224,394		193,366		225,670		148,586		36,417		879,999

Allocation of Central Funds by GOLD Component (December 2003)

			GOL Component		$ Thousands
					00/01		01/02		02/03		03/04		04/05		05/06		Total
	GOL Funds Allocated as of December 2003	Gateway & Clusters	5,341		23,945		16,400		17,745		16,855		15,000		95,286
	GOL Funds Allocated as of December 2003	On-Line Services	8,672		89,216		46,805		57,900		40,821		16,917		260,331
	GOL Funds Allocated as of December 2003	Secure Infrastrcture	32,490		107,299		115,731		136,900		79,410				471,830
	GOL Funds Allocated as of December 2003	Policy & Standards	5,063		3,934		13,000		13,000		11,500		4,500		50,997
	GOL Funds Allocated as of December 2003	Business Case					1,430								1,430
	GOL Funds Allocated as of December 2004		51,566		224,394		193,366		225,545		148,586		36,417		879,874

Appendix D: Principal Documents Consulted

TVO Methodology: Valuing IT Investments via the Gartner Business Performance Framework, Gartner Inc. (no date)

Value for Money is Not Enough in Public Sector IT Projects, Gartner, Inc., 2003.

How to Measure the Public Value of IT, Gartner, Inc., 2003

New Performance Framework Measures Public Value of IT, Gartner, Inc., 2003

E-government Benefits Study. Canberra, Commonwealth of Australia, Australian Government Information Management Office, 2003

Demand and Value Assessment Methodology, Australian Government Information Management Office, 2004

ICA Country Report: Australia. International Conference for Information Technology in Government Administration (ICA) 37th Conference, 2003

Value Assessment in E-Business Transformation - Future Challenges for E-Government. King, P., A. McWilliam, P. Flynn and J. Treadwell, Institute of Public Administration Australia ACT Division, Canberra, Australian Government Information Management Office, 2004

Public value as the focus of strategy. Moore, M. H. Australian Journal of Public Administration, 1994

Quality Internet Services for Government Clients-Monitoring and Evaluation by Government Agencies. Nicoll, P. and B. O'Hanion, Australian National Audit Office, 2003

Performance Measurement and Evaluation of e-Government and e-Governance Programmes and Initiatives. Bovaird, T., OECD e-Government Project, Paris, 2002

Public-Value Failure: When Efficient Markets May Not Do. Bozeman, B. Public Administration Review, 2002

Through the Portal: Enterprise Transformation for e-Government, Deloitte Consulting (Global) LCC and Deloitte & Touche, 2000

Toward the Assessment of Public Values. Dillman, D. A. and J. A. Christenson, Public Opinion Quarterly, American Association for Public Opinion Research, 1974

The 2004 e-readiness rankings. IBM Institute for Business Value, Economist Intelligence Unit and IBM Corporation, 2004

E-government evaluation: A framework and case study. Gupta, M. P. and D. Jana , Government Information Quarterly, 2003

Eliciting Public Values for Complex Policy Decisions. Keenly, R. L., D. Von Winterfeldt and T. Eppel, Management Science, INFORMS: Institute for Operations Research, 1990

Weighing eGovernment." (2003). Proudfoot, S. [html document.] Available: http://www.egovmonitor.com/features/proudfoot01.html.

Focusing on public value: Something new and something old. Smith, R. F. I., Australian Journal of Public Administration, 2004

Measuring the Performance of E-Government. Stowers, G. N. L., IBM Center for The Business of Government, 2004

E-Government vs. E-Governance: Examining the Differences in a Changing Public Sector Climate. Riley, T. B., The Commonwealth Secretariat and, Public Works and Government Services Canada, 2003

Citizens First. Erin Research Inc for The Institute for Citizen Centred Service & The Institute of Public Administration of Canada, 2003

Denying Public Value: The Role of the Public Sector in Account of the Development of the Internet. Rogers, J. D. and G. Kingsley, Journal of Public Administration Research & Theory, 2004

Creating Public Value: An analytical framework for public service reform Strategy Unit, Cabinet Office, United Kingdom, 2002

Internationalization of Evaluation, Picciotto, R. United Nations, 2004

World Public Sector Report 2003: E-Government at the Crossroads, United Nations, 2003

Evaluation and Benchmarking of e-government at the Regional Level: a Pilot Study – STAR (Socio-Economic Trends Assessment for the Digital Revolution), Issue Report No. 35, 2003

Evaluation and Benchmarking of e-government: Status and Perspectives, STAR Issue Report N.34, 2003

Top of the web - User Satisfaction and Usage Survey of eGovernment services, European Commission, 2004

High Payoff in Electronic Government: Measuring the Return on E-Government Investments – U.S. General Services Administration, 2003

Gauging e-government: A report on implementing services among American cities. Kaylor, C., R. Deshazo and D. V. Eck, Government Information Quarterly, 2001

Expanding E-Government Partnering for a Results-Oriented Government – Office of Management and Budget, Executive Office of the President of the United States, 2004

International E-Government Surveys: "E-Government Faces Obstacles". Government and Technology Partnerships, Information Technology Services Branch, Public Works and Government Services Canada, 2004

ICA Country Report Canada, Building Capacity to Accelerate Service Transformation and e-Government ; International Conference for Information Technology in Government Administration (ICA) 37th Conference, 2003

From Vision to Benefit eGovernment Solutions Study: The New Brunswick Case - A Study for CGI and Microsoft - Final Report, IDC Canada Consulting, 2003

Opening the E-Government File: Governing in the 21st Century – Results of the Crossing Boundaries Cross-Country Tour, Centre for Collaborative Government 2001

Measuring Quality of Life: The Use of Societal Outcomes by Parliamentarians. Bennett, C., D. G. Lenihan, J. Williams and W. Young, Centre for Collaborative Government, 2001

Government On-Line 2005 From Vision to Reality... and Beyond, Government of Canada, Public Works and Government Services Canada, 2005

Government On-Line 2004, Annual Report, Public Works and Government Services Canada, 2004

Government On-Line 2003, Annual Report, TBS, 2003

Government On-Line and Canadians: Overview Report, Annual Report, TBS, 2002
Report of the Auditor General of Canada to the House of Commons, Chapter 1, Information Technology: Government On-Line, 2003
Public Policy Forum Report: The Governance of Government On-Line, 2001

Clients Speak – A report on Single-Window Government Services in Canada, The Institute of Public Administration of Canada, 2002

A Framework for Government On-Line: Tier Two Guidelines, TBS, 2000

Preliminary Report of the Government on-Line Advisory Panel, Letter to the President of the Treasury Board, 2002

Transforming Government to Service Canadians Better, Report of the GOL Advisory Panel, 2002

Connecting with Canadians: Pursuing Service Transformation, Executive Summary, Final Report of the GOL Advisory Panel, 2003

Risk Management for Government On-Line, Office of the Auditor General, 2003

Decks, Minutes of Meetings and Presentations to and from TIMS; 1999- 2003

Accenture Performance Measurement for the Government On-Line (GOL) Initiative – Treasury Board of Canada Secretariat, Chief Information Officer Branch, For Discussion, 2002

Performance Measurement for the Government On-Line Initiative – Treasury Board of Canada Secretariat, 2004

Measuring e-Government: "Value for Investment" – GTEC Workshop, 2003

Measuring ROI in E-Government – Bank One (no date)

GOL Service Progression Model, 2002

Results-Based Management and Accountability Frameworks: New 2005 Guidance, Departmental Information Session – Treasury Board of Canada Secretariat, 2005

Strategic Infrastructure Services Sector Planning Approach – PWGSC Information Technology Services Branch, 2004

Secure Channel Project: A Business Plan in Support of the Effective Project Approval Submission to the Treasury Board of Canada – Public Works and Government Services Canada, Information Technology Services Branch, 2004

Appendix E: List of Individuals Who Contributed

Abele, Victor, Senior Director, Policy and Service Transformation, Assistant Secretary’s Office, TBS

Akerley, Marj, Director, Work and Resource Optimization Approach, Information Management and Technology Services,

Albert, Joe, Service Solutions Manager, Infrastructure Requirements & Program Alignment, PWGSC

Armstrong, Sheril, Senior Director, Specialist Support and Shared Practices, PWGSC

Batista, Dan, Director Business Gateway, Industry Canada

Beaudoin, Suzanne, Manager, Canada Site, PWGSC

Butler, Paul, Portfolio Director, Information Technology Management, Consulting and Audit Canada, PWGSC

Christie, Ian, Deputy Director, Innovation – Gateways & Enquiries Services, Foreign Affairs Canada

Danagher, Dan, Director, Internet Services, Foreign Affairs Canada

d’Auray, Michelle, Deputy Minister, Canada Economic Development

Davis, Carmen, Director, Organizational Readiness Office (ORO), Information Technology Services Branch, PWGSC

Fine, Ed, Executive Director, Organizational Readiness Office (ORO), Information Technology Services Branch (ITSB), PWGSC

Florakas, Nick, Director General, Information Management and Technology Services, Agriculture and Agri-Food Canada

Fulcher, Bruce, E-Services Planning, Statistics Canada

Gauthier, Simon, IT Community Champion, Deputy Chief Information Officer, TBS

Hayes, Paul, Director General, Strategic & Client Services, ITSB, PWGSC

Hudon, Larry, Business Manager, Business Gateway, Industry Canada

Jones, Sean, Manager, Strategic Planning, Human Resources and Skills Development Canada

Lebel-Ducharme, Monique, Senior Director, Strategic Communications, TBS

Lemieux, Marie, Director, Strategic and Business Planning, TBS

Lindsay, Dolores, Team Leader, ADM’s Office, Social Development Canada

MacDonald, Robert, Director, ADM’s Office, Social Development Canada

Mantagaris, Elena, A/Director, Gateways and Clusters Directorate, PWGSC

McCallion, Kathryn, ADM, corporate Services, DFA

Mizra, Imran , Director, Project Management and Common Infrastructure, Strategic Infrastructure Services, PWGSC

Molinski, Chris, Co-Chair of IM Leadership Initiative (IMLI), Director General, Technology and Information Management Services, Transport Canada

Moritz, Rita – Champion, IM Community, Director General, Information Management Branch, Corporate Services Sector, Natural Resources Canada

Nadler, Michael, Director GOL Projects, Industry Canada

Patterson, Rod, Director, Information Technology Community, Organizational Readiness Office (ORO), Information Technology Services Branch, PWGSC

Slater, Barbara, Assistant Commissioner, Canada Revenue Agency

Somers, Darryl, Senior Analyst, Gateways and Clusters, PWGSC

Surette, Ron, Champion Service Delivery (SD) Community, Director General, Business Intelligence and CIO, Atlantic Canada Opportunities Agency

Swinwood, Paul, President, Software Human Resource Council (SHRC)

Taylor, Marilyn, Executive Director, Knowledge Services Secretariat, Natural Resources Canada

Thompson, Dave, Director Canada On-Line Services, PWGSC

Tremblay, Hélène A, Senior Project Coordinator, Government On-Line Initiative, PWGSC

Turner, Debora, Program Manager, Knowledge Services Secretariat, Natural Resources, Canada

White, Cathie, A/Manager, Integrated Business Solution Team, Integrated Business Solutions, Farm Financial Programs, Agriculture and Agri-Food Canada

Wood, Donna – Service Delivery (SD) Champion, Director General, Government Information Services Branch, Public Access Programs Sector, PWGSC

Zommo, Mostafa, Director General, Heritage Canada

Zongora, Catherine, Director, Information Management Community, Organizational Readiness Office (ORO), Information Technology Services Branch, PWGSC

Appendix F: Evaluation Options – Costing by Data Source

The following section presents the data sources for each of the evaluation issues of the Evaluation Matrix together with an estimated cost by Option for each data source.

Methodology 1: Interviews

• Useful as a primary source of evidence for Issues: 1, 2, 5, 7, 9, 10.
• Activities include: identifying interviewees, scheduling, developing interview guides, conducting interviews, writing-up notes and analysing findings.
• Option A: About 65 interviews all within the federal government at an estimated cost of $65,000
  • 45 Interviews with community of interested parties identified in the Evaluation Framework; and
  • 20 interviews with senior bureaucrats (DMs and ADMs) in GOL participating and non-participating departments/agencies.
• Option B: About 175 interviews including knowledgeable informants from outside the federal government at an estimated cost of $170,000
  • 45 interviews with community of interested parties identified in the framework development;
  • 35 interviews with DMs and ADMs in GOL participating and non-participating departments/agencies;
  • 30 interviews with GOL leads;
  • 10 interviews with selected stakeholders, such as Advisory Panel members, some Ministers/Parliamentarians and representative from foreign governments;
  • 30 interviews with representatives of other jurisdictions (provincial/territorial/municipal); and
  • 30 interviews with interest groups representing users.
• Option C: About 175 interviews including knowledgeable informants from outside the federal government at an estimated cost of $170,000 Special attention will be paid to seek out and include interviewees with knowledge and expertise in the value and measurement of e-services and e-government in the national and international arenas. Academics, policy makers and others will be included.

Methodology 2: Review, Analysis and Reporting on Existing Performance Measures

• Useful as a primary source of evidence for Issues: 3, 4, 6, 8, 10, 11.
• Option A: estimated cost $30,000
  • Only the main accepted published performance measurement reports will be considered, such as the TBS PMAF, the GOL Annual Reports and Accenture Reports; and
  • The project team does not analyse the original data and underlying methodologies, but uses the findings and conclusions presented in these reports to address the Evaluation Issues.
• Option B: estimated cost $75,000
  • All available GOL performance measurement data, including non-published measures, are sought including performance measures gathered and managed by participating and non-participating departments and agencies;
  • If too much data are available, a decision to review samples from large, medium and small participating and non-participating organizations could be made.
• Option C: estimated cost $75,000
  • All of Option B plus some effort is diverted to research and use POR and other international studies which measure Canada’s and other jurisdictions’ e-government outcomes against inputs.

Methodology 3: Case Studies

• Using case studies affords the project team the opportunity to review a number of specific GOL projects and endeavours in greater detail, thus, supporting the assessment of Issues 2, 4, 6, 8, 10 and 11.
• Option A: estimated cost $150,000
  • 10 case studies representing a cross section of the GOL components and multi-jurisdiction and client centric type of endeavours;
  • Given the small number of endeavours or projects of any type, it will be difficult to generalize findings especially with respect to Issues 6 and 11.
• Option B: estimated cost $150,000
  • 10 case studies as per Option A.
• Option C: estimated cost $300,000
  • 20 case-studies representing a cross section of the GOL components as per Option A and B, as well as case studies of departments and major programs to evaluate the impact of their GOL involvement across components and on other service delivery aspects of their activity, and case studies on entities or endeavours that are of particular interest nationally or internationally due to their work in value determination, public good determination and ROI measurement;
  • Evaluation findings and conclusions would be better supported and lessons learned would be more generalizeable with this broader range of case studies. The ROI question would be explored in depth in several case studies. International case studies would be included for comparison purposes with the Canadian experience.

Methodology 4: Development and Conduct of a Survey of Federal Departments and Agencies

• Useful as a primary source of evidence for Issues: 1, 2, 4, 5, 6, and 8. It would be next to impossible to properly assess Issue 11 without some form of survey of departments and agencies;
• Hardcopy and/or electronic surveys would be sent to heads of departments and agencies; and
• Completed surveys would be returned within four to six weeks.
• Option A: estimated cost $70,000
  • Two different surveys would be conducted;
  • 25 to 30 organizations representing large, medium and small departments /agencies would be sent a detailed survey of about 20 questions that would seek information on client demand, client value as well as benefits to the organization in terms of increased resource efficiencies (savings) or costs avoidance, user benefits, social benefits and contribution to other government priorities; and
  • 25 to 30 organizations also differing in size would be sent a basic survey comprised of about 10 to 12 questions seeking information on user and organization benefits.
• Option B: estimated cost $150,000
  • All 100 or so federal departments and agencies would be asked to complete the detailed surveys.
• Option C: estimated cost $150,000
  • As per Option B.

Note: If an evaluation is to be conducted in 2005-06 fiscal year, it is suggested that the survey preparation, administration and conduct begin in the spring/summer of 2005. A draft survey questionnaire is included as Appendix E of this report. Survey data collection methodology (e.g., web-based, mail, in-person) is to be determined.

Methodology 5: Stakeholder Forums

• Stakeholder Forums would be arranged to discuss themes that are similar to the questions used in the interviews. In addition to supporting issues 1, 2, 4, 5, 6, 7, 8, 10, and 11 it would be possible to use breakout groups to tackle specific issues/questions that have not been fully addressed through other sources, e.g., identifying lessons learned.
• Option A: estimated cost $15,000
  • This option would entail a One-Day Forum of GOL Leads from all participating departments.
• Option B: No Stakeholder Forums; estimated cost N/A
  • Option B relies on the other methodologies for data collection. This presents some increased risk that qualitative input would not be as robust as would be desired.
• Option C: estimated cost $30,000
  • This option entails two separate One-Day Stakeholder Forums. One Forum would be held for GOL/Service Delivery leads in the federal government, the second could bring together representatives from other jurisdictions or from outside government. Part of the Stakeholder Forums would deal specifically with ROI.

Appendix G: Methodology for the Measurement of GOL Cost-Effectiveness

All programs aim to produce benefits that outweigh their costs. Peter Drucker ¹³ suggested that efficiency is about "doing things right" and effectiveness is about "doing the right things". In Government On-line (GOL) terms, doing things right means achieving the optimal relation of inputs (e.g., upfront and on-going costs, leadership/policy, horizontal and inter-jurisdictional coordination, management focus, expertise) and outputs or outcomes.

GOL provides a foundation that has driven and is now being leveraged, enabling new initiatives, leveraging investments, supporting the take-up in e-government. For example, a department that developed an on-line service without GOL specific funding is taking or will take advantage of the foundation put in place by GOL investments. The objective of this methodology is to provide a consistent way to measure the ROI of GOL and e-government across all relevant programs and services.

As the diagram above indicates, the foundation includes the projects and endeavours invested in that drive e-government, such as various Proofs of Concept: Pilots; Leadership; Infrastructure; Regulations; Legislation; Governance; Shared Service Solutions; Common Look and Feel; Accessibility; and, Bilingual Documentation.

Many of the benefits of e-government can be attributed to GOL foundation projects and endeavours. For example, if a department manager invested in an e-government project without GOL funding, they leveraged the foundation provide by GOL. If it was not for GOL, there may have been 2-3 secure channels or 100 disjointed web sites. The Secure Channel investment may have exponential take-up in 1-2-5 years as more and more services involving data that requires enhanced security use Secure Channel services.

Challenges
Key challenges that inhibit adequate value analysis of IT / e-business evaluation are identified below:

• There is no comprehensive methodology available to determine e-government value. There are various forms of value assessment that have been conducted, including measuring improved service delivery, benefits of channel displacement (from phone and counter to the web and lower transactions cost), reduced paperwork, filing, printing and copying, earlier deposits of funds and cash savings to the Crown, revenue generating opportunities from disposal of assets, increasing satisfaction with government services, reducing paperwork and burden of government bureaucracy. Good value analysis models need to include all of these types of benefits, properly categorize them, isolate their drivers, key metrics, and short-term, medium-term, and long-term impacts and effects;
• The benefits and costs of a program or an initiative often cannot be easily expressed in dollars. It can be very difficult to place dollar values on e-business or IT or for that matter GOL programs and services. But this is not unique to e-business or IT. For example, placing value on educational results, health results (the value of human life or its quality), or equity and income distribution results all pose challenges;
• There are limited resources to focus on measuring, tracking and drawing relevant linkages (attribution) of IT / e-business to strategic priorities;
• Benefits too often appear obvious, yet remain elusive to measure;
• Funding oftentimes becomes available regardless of the weaknesses inherent in many business cases. In the absence of sound economic justification, many projects proceed regardless; and frequently,
• Performance measurement systems are not in place or adequate data are not available.

Possible answers to this lie in identifying multiple lines of evidence to support reason and logic, supported by evidence gained from rigorous analysis. The proposed methodology addresses many of the challenges identified above.

An ROI Cost-Effectiveness Primer

The TBS guide on program evaluation ¹⁴ identifies cost-benefit and cost-effectiveness analysis as two common methods used to address the program evaluation requirement for measuring cost-effectiveness.

Definitions

Cost–benefit analysis (CBA): CBA compares monetary benefits (outputs or outcomes) with investment costs (inputs).

• In the case of cost-benefit analysis, program benefits are transformed into monetary terms and compared to program costs.
• Cost-benefit analysis is a method of evaluating the relative merits of alternative public investment projects in order to achieve efficient allocation of resources. It is a way of identifying, portraying and assessing the factors that need to be considered in making rational economic choices. It is not a new technique. In principle, it entails little more than adjusting conventional business profit-and-loss calculations to reflect social instead of private objectives, criteria, and constraints in evaluating investment projects.

Cost–effectiveness analysis (CEA): CEA compares benefit results (outputs or outcomes) with costs, but results are not necessarily transformed into monetary units.

• In cost-effectiveness analysis, program results in some non-monetary unit, such as lives saved, are compared with program costs in dollars.
• For example, the results of a GOL e-service transformation initiative could be expressed in cost-effectiveness terms, as “each $1,000 of program dollars (cost data) results in an average increase of 5% in service delivery improvement for users by using the Internet channel when compared to phone/call centre or in-person walk-up at the counter.
• Or, in an unrelated education context, each $1,000 of program dollars (cost data) results in an average increase of one reading grade (results data). In cost-effectiveness analysis, benefits (or effects) are expressed on some quantitative scale other than dollars.
• Another way to look at this: An initiative is efficient if it maximizes the output/input ratio. The numerator/denominator equation being "the return" (numerator) on the "investment" (denominator).

Nine-Step Methodology

The following is a proposed nine-step methodology aimed at taking a holistic approach to assessing overall GOL and e-government value. Each step in the methodology is identified below:

Step1: Review Literature

• There is an emerging international focus on the analysis of the value of IT, e-business and e-government. This review will isolate issues and approaches relevant to an e-government measurement context. The research will facilitate the appropriate benefits identification and selection activities in the next step.

Step 2: Establish Cost and Benefit Categories

• This step involves establishing cost and benefit categories to be analyzed in the later steps. In identifying the costs and benefits, the approach is to cost all of the GOL funded investments, while taking into consideration all of the broader e-government benefits.

Establish Cost Categories
This step requires identification of categories of costs incurred by government departments and agencies in implementing their GOL projects and endeavours. Analysts will look at and analyze cost data that are made available. The analysis involves identifying tangible or intangible costs. Examples of costs include: fixed costs, hardware acquisition, software development, equipment, hiring and training. Traditional cost categories include upfront and on-going costs. Each of these categories is then further separated into different cost sub-categories depending on the type of project.

Establish Benefit Categories
This methodology suggests using four potential value or benefit categories outlined below.

• 1. Department, agency and federal government benefits: Driver? Reducing costs of government operations, reducing redundancy, consolidating and integrating government systems, enhancing revenue collection and compliance;
• 2. Business and NGO benefits: Driver? GOL Partners’ enhanced competitiveness; reduced amount of regulatory paperwork required for doing business and reduced time spent looking for information, thereby contributing to economic development and competitiveness;
• 3. Service benefits for citizens and society at-large: Driver? Social impacts (e.g., ease of use, take-up, more information "at hand", better customer service, quicker service response, turnaround, client satisfaction, citizen engagement); and
• 4. Contribution to other broad government objectives: Driver? Macro-level improvements in economic, social and technological development, fostering democratic principles, enabling the free flow of information, facilitating transparency and accountability in government, engendering trust in government.

Each benefit is explained in more detail below:

1. Department or agency and federal government benefits:

• automating routine government processes and eliminating paperwork, printing and mailing costs, check processing, document storage and retrieval, and the need for personnel to interact with citizens in person or on the telephone.
• reducing the need for staff intervention in on-line programs that provide government forms, and those that permit citizens to obtain and renew licenses and registrations, apply for government grants and benefits, and get information over the Internet.
• maximizing use of shared and common services: Integrate systems and databases, enabling government to operate more responsively and more efficiently.
• reducing the cost of government operations and enhancing revenue collections. Improved efficiency, higher-value shift of FTE’s to more strategic type of work.

2. Business and NGO benefits:

• contributes to economic growth and productivity, and international competitiveness.
• boon to businesses, both small and large, thereby increasing their contribution to the local economy.
• reduces the amount of regulatory paperwork required for doing business in a community.
• permits on-line document searches and filings, tax and wage reporting, employee background checks, workers’ compensation claims, and one-stop regulatory submissions.

3. Service benefits for citizens and society at-large:

• Measuring the degree to which government information is readily available and on-line services are accessible. Benefits delivered by these programs are defined in terms of convenience, time-savings, quality and completeness of information.
• Reducing administrative burden to citizens, improving client satisfaction.
  • Providing inter-jurisdictional benefits from harmonization of services.

4. Contribution to other broad government objectives:

• Free flow of information
  • Internet facilitates transparency, accountability, and enables a more democratic, inclusive and engaged citizenry.
  • Increases the accessibility of government at all levels, encourages citizen education and participation.
  • Increases trust in government
• Macro-level improvements in economic, social and technological development
  • Greater take-up of ICT (information and communications technology) in the information economy
• Fostering economic development by promoting the beauty, benefits, historic and recreational areas for potential visitors, businesses and residents.
• Enables Canada to become the most connected nation, builds more knowledgeable workers, better access and connection to all Canadians across the regions.
• Demonstrates secure capacity to interact online displaying modern, responsive, transparent government

These benefits have metrics associated with each (e.g., percentages, ratios, dollar value) that will be used in the analysis of the later steps to identify hard and soft, direct and indirect, and tangible and intangible benefits.

Step 3: Conduct Departmental Survey and Review Existing Data

This step involves identifying and gathering data from both the GOL office and via a survey of federal government organizations/departments/ agencies that have and/or have not participated in GOL. The survey is intended to be sent to the departmental GOL Leads.

• Benefits and costs are attributed to all of these groups. There is an opportunity to complete the tasks of gathering data and conducting a departmental survey in parallel. The existing data can be analyzed immediately, while the internal survey is being conducted. This will allow the project to proceed in a timely manner.

Conduct Departmental Survey
This step involves creating a department and agency survey to capture data that are not presently available in existing documentation. This is the most efficient way to collect the data. Refer to Appendix H for a list of potential survey questions for the GOL Leads.

• Because it is comprehensive, a briefing / orientation session is required for each medium to large size department,
• For small departments and agencies, a briefing / orientation guide would be required as well. The purpose of this step is to reduce the burden of work on department/agency representatives.

Review Project Submissions and Review GOL-Funded Projects
This step involves gathering existing data from projects that submitted and received GOL funding. This would include program reviews, original project documentation, post-implementation reviews (PIRs), evaluations, audits, case studies, from all GOL funded programs and services from 2000-2005.

Step 4: Complete Case Studies

This step involves completing case studies of selected projects. Criteria need to be established to select appropriate case studies. The approach is to select GOL projects and endeavours, involved departments/agencies or other Canadian or international entities (number and selection will be dependent on Evaluation Option; see Section 6of the report) that identified themselves in the survey, project reviews or literature review as demonstrating a balanced mix of hard and soft, direct and indirect, and tangible and intangible benefits and costs, or new approaches to e-service and e-government value determination.

Step 5: Identify GOL Projects/Endeavours Objectives

This step involves identifying project objectives and determining what progress or achievements have been made against those objectives (e.g., convenience, access, efficiency).

• Both baseline values and current measures need to be made available to the project team for comparison. It is necessary to establish a baseline of the existing process before transformation. This baseline is compared against the post-transformation state to determine the benefits of the change.
• The objectives will be assessed from the point of view of what achievement or progress was made with respect to the objectives of each project.

Step 6: Classify Programs and Services

This step in the evaluation will also look at classifying all programs and services in logical groupings. The approach to categorization is to identify commonalities within the programs/services so that they can be naturally grouped. The rationale is to select a subset of the programs/services that are representative of the whole, so that data analysis is manageable. Assessments could be completed based on the following dimensions:

• Size of department (i.e., small, medium, large departments)
• Target groups (i.e., citizens, business, non-governmental organizations, international stakeholders)
• Nature and scope (i.e., information services, transaction services)

Step 7: Analyze Benefits

• This step involves assessing the benefits of each of the GOL programs and services and mapping them to one of the four categories of benefits. This requires that every identified project objective be associated with a particular benefit category. This alignment ensures that all project results can be collected and measured within a consistent methodology.
• For example, many GOL programs have reported an increase in user satisfaction. This benefit maps to the “Service Benefits for Citizens and Society at-Large“ benefit category. Benefits such as a reduction in cost for paper, photocopying, and printing or a reduction in counter staff would fall under the “Department or Agency Benefits” benefit category.
• It is important to note that this benefits analysis applies to both users and departments. The benefits of the GOL investment is not just accrued to the internal departments, but also to the external public. The benefits are seen to impact communities, business, and society.

Step 8: Analyze Costs

This step requires identification of tangible and intangible costs incurred by a government department or agency for which data will be made available. This evaluation will only use readily available data sources. Examples of costs include: fixed costs, hardware acquisition, software development, equipment, hiring and training.

• For example, if a sample visit to a counter/office costs $30, the data need to be clearly identified by the department and all the specific costs that make up this figure must be identified as well. This is difficult and therefore data that are actually available will be relied upon.
• The cost analysis involves separating costs into their respective categories (upfront and on-going). Each of these categories is then further separated into different cost sub-categories. For example, depending on the type of project, the Upfront costs category could contain the following cost sub-categories: (i) Application Software and Services; (ii) Environment; (iii) Transition; and (iv) Maintenance.
• The costing analysis also captures any “in-kind contributions” that may have been contributed by external organizations. Any government on-line investment that was matched by non-governmental organizations should be identified. For example, Environment Canada provides a portal for the transportation management of hazardous waste, which may have been developed with funding from both industry stakeholders and government.

Step 9: Conduct Roll up Analysis

The evaluation will focus on the four key benefit areas and the key cost categories. The analysis will examine the achievement of results across the GOL Initiative with respect to cost-effectiveness. This analysis will show to what extent both benefits and costs were accrued under each category.

Appendix H: Sample Departmental Survey

Introduction:
As with any government program or initiative, the value of the Government On-Line (GOL) Initiative specifically, and e-government in general, is in the benefits it delivers to the public and the new avenues it opens to create value. The overall objective of this study is to measure the cumulative return on investment (ROI) of the GOL Initiative over the 2000-2005 time period and e-government across all relevant programs and services. The impact of GOL should be examined from the perspective of its outputs and outcomes in the broader federal e-government context.

Background:
GOL provides a foundation that has driven and will continue to drive future investment and uptake in e-government. For example, a department that developed an on-line service without GOL specific funding is taking or will take advantage of the foundation put in place by GOL investments.

The belief is that there is a relationship between investing in -- and building -- a GOL foundation that enables and supports the development of e-government as a whole over the past five years and into the future. The approach is to quantify many of the benefits of e-government and test how they can be attributed to GOL Initiative and its projects and endeavours in such areas as: Proofs of Concept; Pilots; Leadership; Infrastructure; Regulations; Legislation; Governance; Shared Service Solutions; Common Look and Feel; Accessibility; and, Bilingual Documentation.

Survey:

1. General Profile
   • 1.1 Were you a participant in the GOL Initiative? If so, how did you participate? On which GOL projects?
   • 1.2 Did you have other e-government projects outside of GOL? Were they (GOL or e-government in general) government-wide and client-centric? Were these projects impacted and influenced by GOL foundation projects and endeavours? Please identify which ones and how.
2. Relevance
   • 2.1 Please identify all GOL projects that you were directly or indirectly involved in, year-by-year. We would like to review some of your e-government project submissions. Please provide a list of each project and a key contact for us to follow-up with?
   • 2.2 What are the issues, challenges, needs or problems to which each project was intended to respond?
   • 2.3 Was each project consistent with needs and priorities of targeted stakeholders? Stakeholders would come from your internal department/agency, NGO external organizations, citizens/business, other government departments, international focus.
3. Perceptions and Attitudes
   • 3.1 What are your perceptions of value and benefits received from GOL?
   • 3.2 In your opinion, was GOL an initiative widely adopted within your department/agency? Please elaborate and provide examples and any supporting data.
4. Achievement of Results
   • 4.1 We understand that many project profiles are available on the web site. We would like to review the key departmental reports, program reviews, evaluations, post-implementation reviews (PIRs), audits, case studies, and related material of your GOL projects, year by year from 2000-2005. Please provide a list of each GOL review/evaluation conducted (either done internally or externally).
   • 4.2 What are the benefits that your projects and endeavours have achieved? What progress is being made toward achievement of results? Please provide both actual vs. intended results and any unintended outcomes.
   • 4.3 What are the costs (upfront and on-going) of your GOL funded programs and services?
   • 4.4 Did any projects result in channel migration or displacement? For example, did you see increased take-up of the Internet channel, while seeing a reduction in the amount of counter traffic? Please provide supporting data.
5. Cost–Effectiveness of Results/Outcomes
   • 5.1 Is the relationship between costs and results/outcomes reasonable? How do the costs compare to relevant benchmarks?
   • 5.2 Did actual expenditures correspond to planned expenditures, or was there significant variance?
6. Program/Service Maturity Progression
   • 6.1 Government On-line has developed a Service Maturity Model (see below). This model is identified in the Performance Measurement Accountability Framework (PMAF) for the Government On-Line Initiative (http://www.cio-dpi.gc.ca/si-as/performance/performance_e.pdf). Please indicate each of your GOL project’s progress with respect to the initial, current and future target levels of service maturity for both Information Services and Transaction Services. The following diagram below provides an example.
   
7. External Organization Involvement
   • 7.1 Did any of your GOL projects involve non-governmental organizations or industry?
   • 7.2 If so, how much money was spent by government and how much was “matched” as in-kind contributions by non-governmental organizations or industry?
8. Harvesting Benefits and Managing Costs
   • 8.1 How well did your department/agency achieve the benefits set out in your original submission? Please provide all data and calculations used to support your answer. Include all relevant data points, e.g., number of users, organizations, stakeholders impacted, project budget and financials.
   • 8.2 How is your department realizing the benefits set forth in your government on-line projects? What infrastructure is in place to ensure that expected benefits are harvested? Will results/benefits continue?
   • 8.3 Do you anticipate delivering new services on-line because of the GOL foundation components in place now? If so, what are they?
   • 8.4 Are you aware of any benefits that may have accrued to your partners/contractors on these projects, e.g., able to market expertise gained in the project to other governments/clients/jurisdictions? Please elaborate.
9. Service Delivery Transformation Journey
   • 9.1 What lessons learned and best practices can be attributed to each project?
   • 9.2 How has each project embedded learning and skills development in the organization? How many project leaders have increased their skill set as a result?
   • 9.3 What are the specific next steps that are planned with regards to each project?

1. Performance Measurement for the Government On-Line Initiative, Treasury Board Secretariat, 2004 [back]
2. 1999 Speech from the Throne, Government of Canada [back]
3. 2001 Speech from the Throne, Government of Canada [back]
4. Strategic Directions for Information Management and Information Technology: Enabling 21st Century Service to Canadians, Government of Canada, 1999 [back]
5. quoted from Strategic Directions for Information Management and Information Technology: Enabling 21st Century Service to Canadians, Government of Canada, 1999 [back]
6. These services could include exchanges and transactions between users and the government, and dealing with regulatory matters. [back]
7. For purposes of brevity, OGDs will be used to refer to federal departments and agencies, including their respective programs. [back]
8. GOL On-Line 2004, Annual Report, Public Works and Government Services Canada, March 2004 [back]
9. http://www.gol-ged.gc.ca/pnl-grp/reports/final/final00_e.asp [back]
10. http://www.cio-dpi.gc.ca/si-as/performance/performancetb_e.asp [back]
11. The March 2005 meeting of the OECD E-Government Network in Paris included presentation on progress toward e-government in Mexico, Norway and Denmark and recent work on "Evaluating the Value of E-Government" for the OECD Public Governance Committee. [back]
12. Performance Measurement for the Government On-Line Initiative, Treasury Board of Canada 2004 [back]
13. http://www.wachovia.com/small_biz/page/0,,447_972_1695_1946_1965,00.html [back]
14. Program Evaluation Guide, Treasury Board Secretariat, 1991 [back]