A formative evaluation of Strategis was undertaken in 2001-2002. A formative evaluation typically focuses on the design and delivery of a program or initiative and provides feedback on areas that are working as well as areas that need to be improved. The purpose of this evaluation was to examine how Strategis has functioned as a web service for Industry Canada and external clients and, further, how it may have transformed the delivery of Industry Canada services. A second objective for the study was to inform decisions regarding the possible future directions for Strategis. In order to meet these two study objectives, this summary presents findings in terms of fundamental evaluation questions regarding the current and past performance of Strategis, as well as in terms of considerations for the future.

Methodology

The evaluation addressed a series of issues categorized according to the Treasury Board Secretariat evaluation guidelines:

Issue 1 Relevance – Is Strategis meeting the needs of the relevant businesses, consumers, and other users in providing them with a source of secure, affordable, accessible, user-friendly, credible, well-packaged, unique information products and services, in order to promote competitiveness and efficiency?

Issue 2 Objectives Achievement – To what extent is Strategis achieving its intended objectives?

Issue 3 Cost Effectiveness/Alternatives – Is Strategis well designed and implemented or are there significant modifications which need to be made to increase its efficiency and effectiveness?

These issues were addressed using sound evaluation practices and multiple lines of evidence for each issue. The methodologies included:

review of documents and Strategis Management Information System (MIS) data;

interviews with Industry Canada staff;

feedback from Strategis Content Providers;

online survey of Strategis users;

interviews with intermediaries and associations;

case studies of nine Strategis products;

a comparison of international sites; and

interviews with representatives of Industry Canada Regional Offices.

Please see Section 1.2 of the Report for more detail on the methodology for this evaluation.

The Current and Past Performance of Strategis

The findings are summarized below according to evaluation issue (relevance, success and cost-effectiveness).

Relevance (Sections 4, 5 and 6 of the Report)

Strategis remains a popular, relevant site. However, recently, usage has levelled off and interviews, document reviews and surveys indicate that in some respects, it is a service which has reached maturity. For example, certain components of Strategis now have several competitors. Strategis appears to need some rethinking regarding its vision, governance and value proposition to users.

Success (Section 5 of the Report)

The original objective of Strategis was essentially to enhance Industry Canada's ability to get information out to its clients. There is no doubt that Strategis has achieved this original objective set in the mid-1990s. The capability and use of the Internet, and Strategis with it, has evolved significantly since then. It is therefore important to gauge Strategis' success in terms of the current context and clients' expectations.

The majority of clients report that they are satisfied with the information and tools on the website and that Strategis has had a positive impact on their business or organization. The most important impact was one of timesaving, followed by its ability to increase reach and access to information. Strategis was also found to be highly trusted by users as a source of information. Conversely, Strategis does not stand out from other similar web products in terms of its ability to satisfy users, especially infrequent users.

Cost effectiveness (Sections 6 and 7)

Strategis has clearly improved the cost effectiveness of certain functions and services of the Department, especially those involving the dissemination of published information and the completion of transactions. However, the effectiveness of Strategis products is hampered by a number of factors. These factors include the increased ability of users to independently search and retrieve information without the involvement of Industry Canada officers, the myriad of independent contributors, combined with a lack of dedicated resources to maintain the system. In addition, Strategis has not appeared to have had a significant impact on the way in which Industry Canada officers interact with their client groups. Finally, the shared accountability of Strategis appears to have led to a significant governance gap further hindering cost-effectiveness.

Future Considerations

The report structures the findings, the conclusions and recommendations by four major themes. These themes start with the broad issues (the relevance and vision of Strategis, the impacts of Strategis and governance issues) and then move to the more specific issues related to the design and delivery of Strategis. The key conclusions and recommendations emerging from the evaluation are summarized below according to these four themes. Please note that the conclusions and recommendations related to vision and governance are likely to be of most interest to senior executives, senior managers and the CIO (Strategic). Those conclusions and recommendations related to client orientation and design and delivery are more to individual product managers and the CIO (Operational).

Vision (Section 9 of the Report)

Conclusion – A common, unifying vision for Strategis is needed. Many Industry Canada interviewees felt that Strategis lacked a common, unifying vision to provide overall direction to the site. This lack of vision or focus was believed to have contributed to the decrease in the emphasis placed on Strategis as an innovative product for the department and the resulting lowered management attention. In addition, in the absence of any concerted vision, individual sites have taken on the responsibility for their own marketing and planning. This has contributed to the continued, yet varied, growth of the site without a strategic focus.

There were divergent views about Strategis within Industry Canada. For example, there was some disagreement over the continuing relevance of Strategis. One group felt strongly that Strategis remains a valuable, well-used tool that serves an ever-growing number of Canadians. The second predominant view was that Strategis is a mature product that has stagnated and no longer meets the needs of users and Industry Canada. In addition, there were differing opinions between those who perceived the value of a common department-wide website or Internet presence and those who valued the independence of the individual products.

Recommendation – Review the future vision and form for Strategis. There is a need to develop an updated guiding vision for Strategis. It is clear that this is an opportune time to pose some key questions about Strategis and its role within the Department. Suggested questions include:

What does Industry Canada wish to achieve with its online presence within Strategis?

What is the value proposition for Strategis? (i.e., What unique value does it seek to continue to deliver, build-on or develop with key client communities? Also see client orientation discussion below)

What is Strategis' role in promoting the use of new technologies?

How does Strategis help Industry Canada implement its legislative mandate and achieve the Strategic Objectives? What role should Strategis play in supporting the Portfolio?

A number of conceptual models are discussed in this report. The intention is to use the results of this evaluation to inform the discussion on future directions for Strategis and for these models to provide a framework for this discussion. These models address concepts such as resources versus reach (see page 13), diversity and complexity (pages 16 to 17), continuum of involvement (page 55) and the line of visibility (pages 68 to 71).

Governance and Shared Accountability (Section 9 of the Report)

Conclusion – There is a need to engage senior management in the department around the governance and maintenance of Strategis. One of the critical issues facing Strategis is the need to engage senior management of Industry Canada in the governance and ongoing management of Strategis. While the overall responsibility for Strategis lies with the CIO, individual groups are responsible for the content that they deliver and maintain. Currently, there does not exist a department-wide forum or committee at which to discuss strategic issues or overall direction for Strategis. The Strategis Content Providers Committee is used as a vehicle for sharing information about common initiatives with officers and others working with client sites. However, this committee addresses technical rather than strategic issues. If a new vision were to be developed for Strategis, an existing committee at a management level is not already in place. This absence of a governing body for Strategis appears to have contributed to the lack of shared accountability for this Industry Canada-wide initiative.

A significant number of officers, Strategis content providers and publishing staff commented that there was little management support of a vision for an online client service strategy in their organizations. Indeed, in a number of organizations, officers working with online products commented that senior managers did not view working with websites as a competency for professionals, instead viewing it as a junior role.

Recommendation – Reconvene a senior management committee with responsibility for overseeing Strategis. This committee would be responsible for providing overall governance to Strategis and would provide a mechanism for making decisions about Strategis at the senior management level. This committee would be tasked with the development of a new vision and directions for Strategis as well as developing clearer accountability around site management.

Client Orientation (Section 9 of the Report)

Conclusion – Strategis no longer offers a unique value proposition to clients. When Strategis was first launched, there were few, if any, similar services provided by the private sector or other government sites. This no longer true. For example, company directories and industry specific information are available on private sites. There is also an internal perception that Industry Canada itself has a number of competing or similar web products (e.g., Strategis, Business Gateway, Canada Business Service Centres and the Industry Canada corporate website). Information and services on Strategis that are unique include department-specific initiatives as well the delivery of mandated and legislated services.

Conclusion – The information and services on Strategis have an impact on clients. More than one quarter of respondents (26%) reported that Strategis had definitely had a positive impact on their business. In addition, another two-fifths (39%) stated that Strategis had a somewhat positive impact on their business. The most common impact was time savings. Of the total number of clients who answered the question on impacts 41% reported that Strategis had saved them time.

In some respects, the nature of Strategis impacts appears to correspond to the maturity of the Internet services marketplace. As of 2002, it appears that the biggest impacts occur when Strategis saves time in the processing of (somewhat complex) transactions. As time goes on, it may be expected that Strategis and Internet services will show greater impacts in terms of serving needs with greater richness of information and to accommodate higher diversity (see Section 2.3 of this report). Such future impacts will, of course, also depend in Industry Canada's management of these services.

Recommendation – Strategis as a whole must work to ensure a client-centred focus. The uneven client focus across Strategis was a theme that emerged strongly from the fieldwork. This issue is closely related to the development of a new vision for Strategis. Indeed, the need for a strong focus on the client may be addressed by a new vision that includes a client-centred philosophy. Groups developing tools, information or other products for Strategis should be able to identify the primary audience for their site (e.g., For whom are we developing this product? What need(s) will it meet? What change(s) would we like to effect as a result of the tool/information that we are developing?). In many cases, there are secondary audiences for the product as well as peripheral users. In addition, Strategis continues to be difficult to navigate for new and infrequent users. Although there has been some usability testing of the site conducted in the past and changes made, additional work is required to structure Strategis following client centred principles.

Design and Delivery (Section 9 of the Report)

Conclusion – Shop Floor is not currently meeting the needs of all internal clients. Confusion over the role of the Shop Floor and requirements for developing and launching a new product is causing a great deal of friction and frustration among internal clients. The Shop Floor is not seen as a client-service oriented organization and is seen by many to be a barrier. One suggestion made by numerous interviewees was that the Shop Floor should become a service organization that tries to facilitate the work of the content providers rather than be a hindrance as they are currently perceived to be. Rather, it was suggested that they should adopt the role of facilitator rather than "police." Individual staff are well regarded, but internal clients see the structure and process of the Shop Floor as significant barriers to client service.

Recommendation – Review the role of the Shop Floor. In order to better meet the needs of internal clients, the Shop Floor should:

review the needs of internal clients;

assess the role that the shop floor should play to best meet these needs;

revisit the client relationship (e.g., proactive as opposed to reactive);

analyze the required skills and training required for staff; and

promote the exchange of best practices between products and services.

In addition, the Shop Floor should communicate the menu of services available to internal clients and clarify. This review of the role of the Shop Floor would be dependent on decisions made with respect to the vision and future form of Strategis.

Recommendation – Develop site management protocol or tools. Strategis should develop site management protocol or tools. To provide a much more useful service, it was recommended by a number of interviewees and Strategis content providers that a greater degree of rigour should be applied to clearly defining the site scope, content selection, inclusion criteria, information indexing and classification. Common site management protocol or tools could be developed. One suggestion made a number of times was for a simple checklist of responsibilities and timing for site management. Another suggestion was to share best practices across Strategis.

In summary, the vision for Strategis will set the context for governance and client orientation. Design and delivery issues will then be informed or affected by decisions made around the vision and governance. The four levels of issues, conclusions and recommendations are inextricably linked.

Adobe Acrobat Version (PDF - 593KB - 113 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.