Evaluation of Canada's Digital Collections Program
Executive Summary
Audit and Evaluation Branch
February 19, 2003
Executive Summary
Canada's Digital Collections (CDC) program has two central aims:
- To provide young Canadians in all parts of Canada with initial work experience in the multimedia sector; AND promote the development of the Canadian multi media industry and, in particular, position new youth-run enterprises for success in the marketplace
- To provide wider access to Canadian material of public interest via the Information Highway; and demonstrate the productivity enhancement benefits of digitization.
Methodology
This evaluation of CDC program used the following data collection methods to answer key evaluation framework questions regarding the effectiveness and outcomes of the program:
- interviews with 10 key informants of the program as identified by Industry Canada between March and May 2002
- fax survey of 59 contractors (18% response rate) and 42 custodians (16% response rate) between April and June 2002
- telephone interviews with 575 youth participants (response rate of 47%) between March and June 2002
- analysis of data collected from 249 visitors to the CDC web site in 1998
- a document review of on-line materials from the CDC web site.
Rationale for CDC
Key informants generally believe that CDC agrees well with the Connectedness Agenda. Two respondents explicitly said that they felt the CDC program helped shape the Connectedness Agenda, and most think that the program has remained relevant or has increased in the past five years. Several respondents noted that the program's relevance was due to its usefulness as a tool for providing young people with experience and addressing the needs of "at-risk" youth.
Just over half of the contractors (56%) and two-thirds of custodians (69%) surveyed indicated that they received additional funding, resources, or support from other government programs for the purposes of digitizing material. The majority indicated that the support was financial (73% of contractors and 86% of custodians).
Most key informants believe that discontinuing the Digital Collections would have a negative impact. The most common concern was that terminating Digital Collections would deprive young people of opportunities to gain work experience in multi media. Others said that ending the program would harm the heritage-preservation cause, as many of the existing collections would never have been digitized without the program.
At the same time, several key informants pointed out that Canadian Heritage was about to embark on a major program of digitizing Canadian historical and cultural content. They see CDC as having initiated the program and demonstrated the potential for digitizing content. They believe that CDC can safely turn to other objectives.
The majority (54% of contractors and 81% of custodians surveyed) indicated that they were planning other digitization projects. Of those planning future projects, half indicated that they were doing so without any CDC support.
Two-thirds (66%) of youth participants surveyed said that they had found work as a result of their CDC involvement, and 40% said that the CDC project was very helpful in finding the work.
Program delivery
Generally, key informants expressed satisfaction with the level of support received from CDC and Industry Canada. However, one respondent said that while support exists, it is insufficient for their needs. Another was satisfied with the application process and guidelines but indicated that promotional support was lacking.
Most contractors surveyed (89%) agreed that Industry Canada provided their organization with sufficient guidance and information to successfully fulfil its contracts and that custodians provided them with sufficient guidance in terms of material selection and content arrangements (82%).
Youth participants reported that they received training in technical (76%) and business management-related issues (41%). This indicates that a sizable portion of youth participants are learning business skills in addition to technical skills.
Key informants stated that CDC's main web site acts as a gateway because all collections are required to be listed there and also with major search engines, which in turn, helps generate traffic. Bulletins are also issued to librarians and archivists.
A few key informants argued that CDC should be promoted more vigorously. One respondent suggested cross-marketing with the Virtual Museum of Canada as a possible option.
When contractors were asked whether the lack of information on government programs was an important obstacle in their organization's plans for business expansion, half (50%) said that this was not an important issue, 30% indicated that it was, and 20% did not know.
There was widespread agreement among key informants on the effectiveness of the selection process. Respondents praised the clear criteria, the fairness of the process, and the use of adjudicators to settle disagreements.
Results
Key informants reported positive feedback on the CDC program from custodians and contractors. However, a few informants said that the feedback also pointed to needed improvements in the contribution agreement. Improvements were said to be needed because the process has become too bureaucratic.
Of those contractors that did multi media work before becoming involved in CDC, about half indicated that their organization had experienced an increase in multi media activity as a result of being involved in the CDC program.
More than half of contractors indicated that the CDC program had increased their organization's visibility or exposure in establishing new opportunities or alliances with cultural institutions (63%) and new networks at the local level (54%).
More than half of custodians indicated that as a result of the CDC program, they had either established or are considering establishing new opportunities or alliances with cultural institutions (74%), new networks at the local level (60%), and joint ventures with other content providers (62%).
When asked what their main activity has been since the project ended, most participants said that they were either working full-time or part-time (44%), pursuing studies (40%), or both working and pursuing studies (10%). Of those working, the majority (62%) said that it was in an information technology (IT) or multi media-related field. Of those pursuing studies, about two-thirds (65%) were doing so at the university level.
All youth participants were asked if involvement in the project had an influence on their educational path or their career path. Just over half stated that the project had somewhat or a great deal of influence on both their educational and career paths.
A large majority of youth participants also agreed that involvement with CDC through the project made them more aware of career opportunities in the education, heritage, or cultural fields (73%); made them understand what is needed as a professional in the workplace (83%); and increased their understanding of business and entrepreneurship (67%).
Nearly all youth participants surveyed (94%) stated that their experience with the project met or exceeded their needs. Approximately one-fifth stated that they had started a multi media or information technology business (14%) or intended to do so (5%) as a result of their participation in CDC.
Both contractors and custodians were asked to rate the CDC program in terms of its usefulness in the production of new digital materials and the preservation of cultural and historical heritage. For both of these items, more than three-quarters of the contractors and custodians (ranging from 76% to 81%) rated the CDC program as very useful.
Key informants mentioned that the collections are useful from the point of view of promoting Canadian heritage and creating quality Canadian content on the Internet. Respondents mentioned that the content reminds viewers of Canada's history and roots and creates a sense of national identity and self-awareness.
Lessons learned
Key informants cited the following as strengths of the program:
- relatively little red tape, application process is streamlined
- offering training opportunities and good content
- promoting Canadian culture
- creating professional relationships
- helpfulness of CDC personnel.
Key informants also cited weaknesses of the program:
- Payment structure and contribution agreements are irritants.
- Bureaucracy when modifications to project contract terms are required.
- The focus is too narrow; it should fund a wider range of collections.
- Funding is delayed or inconveniently timed.
- Salary and funding levels are too low, especially for smaller organizations, as additional support is required to supervise the project; for many, current staffing levels are insufficient for this task, thus reducing participation by these groups.
- Contractors exploit the program for their own benefit.
Custodians were asked to identify what they thought was the CDC program's greatest positive element. Most respondents identified youth training opportunities (36%) or providing greater access to Canadian history online (36%) as the key positives.
Custodians were also asked to identify what they thought was the CDC program's greatest negative element. Most common issues raised included funding guidelines, per hour requirements, and budget constraints (29%) and bureaucracy, paper work, project administration, and reporting (19%).
Key informants offered the following suggestions for improvement of the CDC program, many of which attempt to address the weaknesses uncovered above.
- More funding should be made available for upgrading existing projects, raising wages, and doing more promotion.
- Reduce paperwork and bureaucracy.
- Content quality, particularly in the application process, should be better monitored.
- The program should target specific youth groups, such as the disabled, and those in inner cities.
- The program needs new objectives or to look into new areas (such as digitizing corporate or community collections), however, corporations should be able to fund their own collections.
Over half (57%) of custodians offered improvements that could be made to the CDC program. The most commonly mentioned improvements were changes to funding/operating guidelines/management (33%) and increased funding/access to funding/broadening of eligible expenses (21%).
Conclusions
The most cost-effective CDC projects have the following attributes:
- strong planning and conceptualization from the outset
- reliance on accurately researched information
- management by an experienced contractor
involvement of youth participants who are highly skilled in web site creation, are highly motivated, and possess a positive attitude and a keen interest in the subject matter.
Recommendation 1. The CDC Program should be maintained
Those involved in the program are generally satisfied with CDC. This is displayed in the fact that most contractors and custodians (86%) rated the CDC program as "good" or "excellent," and nearly all youth participants (94%) said that the particular project they were involved with through CDC met or exceeded their expectations. Key informants mentioned that CDC is not only for training and providing youth with valuable work and business skills, but it also promotes Canadian heritage and creates quality historical content for the Internet.
Recommendation 2. Consideration should be given to two options for broadening the CDC program:
- migrating it to departments/agencies that are more directly involved with culture.
- expanding the involvement of departments, agencies, and even private organizations to increase the range of collections.
However, it is likely that diminishing returns may be encountered in that many archive collections have been digitized. The remaining collections of significance are likely larger, and in some sense, more significant. Digitizing them may well require more sustained resources than can be managed over the short time available to participants in the CDC program.
Currently, formal editing or quality control processes exist and existing custodians appear to have been able to maintain quality control. However, with larger, more significant collections, a need for formal editorial control may emerge.
Recommendation 3. Industry Canada or some other federal department/agency should maintain the "portal" for all collections. This should not preclude links to other sites to increase access to collections.
If the program evolves to a wider group of sponsors or migrates to a few specific agencies, consideration should be given to ensuring that the existing collections are not "lost." A danger exists in creating a diffuse initiative without standards and a wide dispersion.
Recommendation 4. Industry Canada or the future sponsor of CDC will need to allocate resources to maintain and upgrade popular, high quality sites.
Several observers noted that the limited budgets available created a ceiling or "capped" the resolution of the collections. This limited their use to public education and general information and precluded their industry or professional use. At the same time, technological advances in multi media may create a situation where older collections may appear dated, and at some point in the future they may become less accessible as newer viewing technologies become standard.
Recommendation 5. Consideration should be given to awarding a management support fee for smaller organizations with qualified collections. Care is needed to ensure that this fee is reserved for the small, non-profit, or community organization.
A recurrent complaint was that the application process was bureaucratic. Further, for smaller organizations, the cost of making the proposal and managing the process required additional staffing resources. The terms of the Program do not allow for support for organizations to hire additional management support. According to a few respondents, this precluded some organizations with good collections from participating.
Recommendation 6. The department ultimately responsible for CDC should increase its efforts to publicize CDC, thus raising the market profile of the custodians and their collections and generating greater public interest in the collections as an educational/informational resource.
Several key informants noted that the CDC Program is not well known, especially in light of how it could serve the educational market.
Recommendation 7. Industry Canada needs to develop a strategy for ongoing maintenance of the collections.
An important issue in the maintenance of web sites as technology evolves. Overtime, excellent web sites may become obsolete as new Internet techniques become available. This can compromise the investment into the digital collections.
Evaluation of Canada's Digital Collections Program — Final Report
(PDF, 341KB, 57 pages)
Management Response to a Report Entitled Evaluation of Canada's Digital Collections Program (PDF, 78KB, 6 pages)
Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.
Date Created: 2004-01-14
|