Government Documents Reference Service in Canada: Implications for Electronic Access

When phone back situations are included in the category "more than 10 minutes," three out of the four depository types –with the exception of public full depositories– spend 10 or more minutes either at approximately the same rate or at a greater rate with patrons who use the telephone to ask reference questions as with patrons who ask their questions in person. Indeed, public selective libraries display the most consistency in this regard. An almost equal percentage of the time is devoted to telephone questions and to walk-in questions in the time-spent categories of "one to four minutes" and "greater than 10 minutes." Given the data in Table 10, it is surprising that complete or partially complete answers are provided to walk-in questions at a 10% greater rate than to telephone questions, as noted in Figure 16. It would be logical to expect that the efficacy rates would be about the same. One reason for this may be that library staff members may be able to get a better sense of the question when it is delivered in-person.

Another question variable of interest in this study had to do with whether the subject matter of the question dealt with the legislative or the executive branch of government. In total, proxies asked 324 executive questions and 164 legislative questions. As can be seen from Figure 17, complete and partially complete answers were provided to legislative questions at a statistically significantly higher rate than to executive questions (x²=24.92, df=3, p <.01).

While legislative questions were completely or partially completely answered at a rate of 48.2% (79 out of 164 questions), executive branch questions received complete or partially complete answers 39.5% of the time (128 out of 324 questions). Moreover, legislative questions were given referrals at a substantially lesser rate than were executive branch questions. While executive branch questions were referred at a rate of 26.2% (85 out of 324 questions), legislative branch questions were referred only 7.9% of the time (13 out of 164 questions). Table 11 confirms these findings by analyzing the four types of depository libraries and their ability to provide complete and partially complete answers to legislative or executive branch questions.

Table 11: Comparison of Legislative and Executive Questions

Whereas selective depositories provide complete and partially complete answers to legislative and executive questions at about the same rate, the difference between these two types of questions is most readily apparent in public full depositories and academic full depositories. Full depositories provide complete or partially complete answers to legislative branch questions at a rate that exceeds complete or partially complete answers to executive branch questions by 10%-25%. One reason for this phenomena may be that legislative questions are more homogenous than executive branch questions. While the subject matter of legislative questions can be as broad as executive branch questions, the locations in which the answers to legislative questions can be found are limited in number. For instance, once a librarian knows how to find one statute, one bill, or one comment in Hansard, then the answer to any subsequent questions dealing with statutes, bills, or debates will be found in the same location. It should also be noted that the questions themselves often include clear references to appropriate sources, e.g. "Where can I find a bill about Topic X?" Executive branch questions, on the other hand, are heterogeneous not only in regard to subject matter, but also in regard to potential locations.

The present study also wanted to know whether a distinction prevalent in the scholarly literature of librarianship between data-retrieval questions and document-retrieval questions was germane for government-based reference questions. As noted above, Katz (1996) defines data retrieval questions as those in which individuals ask "specific questions and expect answers in the form of data." Document retrieval queries are those in which patrons "want information, not just simple answers," and this information is "usually in the form of some type of document" (p. 18). Figure 18 examines whether depository libraries as a group provided a greater number of complete or partially complete answers to data retrieval questions or to document retrieval questions.

Data-retrieval questions were asked 195 times, while document-retrieval questions were asked 293 times. Complete answers were provided to document-retrieval questions (93 out of 293 questions) at a slightly higher rate (31.7%) than for data-retrieval questions (50 out of 195 questions) (25.6%). When complete and partially complete answers are combined, the rate for document-retrieval questions is 45.4% (133 out of 293 questions), while the rate for data-retrieval questions is 37.9% (74 out of 195 questions). Data-retrieval questions (61 out of 195 questions) (31.3%) are referred at more than twice the rate of document-retrieval questions (37 out of 293 questions) (12.6%). Differences are statistically significant (x²=25.86, df=3, p <.01).

6.5 Referrals and No/Incorrect Answers

The previous sections have concentrated on an examination of the various rates of complete and partially complete answers in different categories. In this section, a closer look is given to the reasons for no/incorrect answers, as well as to the types of referrals made by library staff members. Also presented are comparative data about those questions that were referred and those questions to which no answer or the incorrect answer was provided. In total, 98 questions (20.1%) were referred to various governmental and non-governmental institutions. As Figure 19 shows, fully half the referrals (49 times) were to government departments. Proxies were referred to governmental or legislative libraries (7 times) (7%). Proxies were referred to other non-governmental libraries, usually at a university, at a rate of 29% (28 times). In addition, 14% of the time proxies were referred to external non-governmental agencies or establishments that were not libraries (14 times).

Of the referrals made to government departments, 65.3% were made by public selective depositories. This should not be surprising given that many public selective libraries do not collect a wide array of government documents. More interesting is the fact that both academic full depositories and public full depositories each referred patrons to government departments at a rate of 16.3% despite having in their holdings the full range of official publications. Of the referrals to other non-governmental libraries, 71.4% of the time such referrals were made by public selective depositories and academic selective depositories. Again, this is entirely understandable, given the incomplete nature of the holdings of selective libraries. Yet, 28.6% of academic full depositories also directed proxies to seek out other non-governmental libraries. Referrals may be explained by a desire to provide patrons with the most current information, or it may be that staff members are not fully aware of the resources held by their own library.

No/incorrect answers were received by proxies 183 times, that is, at a rate of about 38%. As Figure 20 demonstrates, the most common explanation for a proxy's receiving a no/incorrect answer was that the library staff member simply did not know how to find the needed information. This happened in 37% of the cases in which a no/incorrect answer was given (67 out of 183 questions). The next most common reason for a no/incorrect answer was that the library staff member provided an answer, which, unfortunately, was inaccurate. This happened 21% of the time that a no/incorrect answer was provided (38 out of 183 questions). Ten percent of no/incorrect answers (18 out of 183 questions) were the result of the unwillingness of library staff members to answer proxy questions – a circumstance which may, of course, mean that library policy precludes providing a certain type of service over the phone, or to those not perceived to be primary clients. Indeed, in the case of telephone queries, where the patron cannot see the selection of available resources, some libraries may have a specific policy to refer telephone questions quickly to a government contact. As a last point, 27% of the time that a no/incorrect answer was provided, proxies were told either to come into the library, phone back, or come back at a more opportune time (49 out of 183 questions). Again, some libraries may have policies which encourage users to come in and decide for themselves which resources best fit their needs, or to work through what a staff member may have thought was a complicated reference interview. Sources were unavailable nine times (5%). Two other reasons occurred.

Staff members at public selective depositories accounted for 52.2% (35 out of 67 questions) of the "did not know" category. More problematic is the fact that staff members at academic full depositories told proxies that they "did not know" at a rate of 20.9% (14 out of 67 questions). Staff at public full depositories accounted for 9% of the "did not know" category (6 out of 67 questions). The type of question that most received a "did not know" reply (46 out of 67 questions) was an executive branch question (68.7% of the time), as opposed to a legislative branch question (21 out of 67 questions) (31.3% of the time).

Staff members at public selective depositories accounted for 47.4% of the "tried but got incorrect answer" category (18 out of 38 questions). Their counterparts at academic full depositories fell into this category at a rate of 21.1% (8 out of 38 questions). Staff at public full depositories accounted for 13.2% of the "tried but got incorrect answer" category (5 out of 38 questions). The type of question that most received a "tried but got incorrect answer" reply (25 out of 38 questions) was an executive branch question (65.8% of the time), as opposed to a legislative branch question (34.2% of the time).

Figure 21 addresses the question of whether the type of depository library had an impact on the rate of referrals and no/incorrect answers. Academic full depositories, public full depositories, and public selective depositories gave no/incorrect answers at approximately the same rate, which hovers between 34% and 36%. Academic selective depositories provide no/incorrect answers at a significantly greater rate – 53.9%. Referrals were provided by academic full depositories, academic selective depositories, and public full depositories at an approximately equal rate ranging from 9% to 15%. Among the four types of depository libraries, public selectives referred patrons at an almost 30% rate. Not surprisingly, those depositories that had full collections of government documents were less likely to provide no/incorrect answers or to make referrals.

Figure 22 provides a detailed look at referrals and no/incorrect answers by geographical area. Depository libraries in Ontario provided no/incorrect answers at a rate of only 25.4% (42 out of 165 questions) and referred patrons elsewhere only 17% of the time (28 out of 165 questions), clearly the best record in Canada. Depository libraries in the Atlantic Provinces gave no/incorrect answers at a rate of 37.3% (28 out of 75 questions), while these libraries referred questions elsewhere 21.3% of the time (16 out of 75 questions). British Columbia is much like Atlantic Canada in this respect. Depository libraries in British Columbia provided no/incorrect answers at a rate of 39.6% (21 out of 53 questions) and referred at a rate of 15.1% (8 out of 53 questions). The Prairie Provinces and Québec had no/incorrect answer rates of 45.6% (41 out of 90 questions) and 48.6% (51 out of 105 questions), respectively. Referral rates for the Prairies and Québec were 22.2% (20 out of 90 questions) and 24.8% (26 out of 105 questions), respectively.

The relatively high levels of no/incorrect answers and referrals revealed in this study are a cause for concern. One explanation may be found in a study conducted by Harris and Marshall (1998). They provide evidence, based on interviews with Canadian library directors, that restructuring imperatives occasioned by budget constraints have led to the increased use of paraprofessionals in reference functions. Since paraprofessionals may not have the necessary background knowledge and specialized skills to answer government documents questions, a certain deskilling of reference work may be taking place. In addition, Harris and Marshall note that library directors envision librarians as primarily managerial material – a circumstance which may result in high-quality reference librarians being taken away from desk duty and placed in exclusively administrative roles. Further study is needed to determine the impact on patrons of no/incorrect responses and referrals. Do patrons become discouraged? Frustrated? Do they follow up on referrals or give-up? Do they turn to other sources for the information that they need?

6.6 Sources Used to Answer Questions

An important aspect of this study concerns the kinds of sources library staff members used to answer government documents reference questions. As indicated earlier, all 15 different questions asked by proxies could be answered using Internet-based Web resources. Dolan and Vaughan (1998) report that, by the end of 1996, 89% of depository libraries had Internet access, and that of the 11% that did not have Internet access in December 1996, some 70% were planning to have such access within one year. When the present study was conducted in December, 1997, it was not unreasonable to suppose that Internet access was available in some 96% of federal depository libraries. Dolan and Vaughan (1998) also report that print sources are used much more frequently in depository libraries than are electronic sources. The present study therefore wanted to determine the extent to which library staff turn to various types of sources to answer patron questions. Figure 23 provides a summary of staff source use.

Print materials constitute the largest single source (45.7%) used by staff members at depository libraries (223 out of 488 questions). The Web alone was used 11.5% of the time (56 questions), and the Web in combination with another source was used at a rate of 5.5% (27 questions). Thus in whole or in part, Web use to answer government document reference questions hovers around 17% (83 questions). About 23% of the time no sources were consulted (112 questions), and in an additional 9.6% of cases (47 questions), the only source used was a library on-line public access catalogue (OPAC). CD-ROMs or databases were used at a rate of 3.7% (18 questions), while microforms were employed at a rate of just over 1% (5 questions).

Can more be said about the use of "no sources?" Of the 112 questions for which no sources were used, 55 were walk-in questions and 57 were telephone questions – an almost equal division. Was there a difference among depository libraries in their use of no sources?⁶ On those 112 occasions when proxies indicated that no sources were consulted by library personnel, 48.2% of the time it was a question of public selective depositories, 30.4% of the time academic full depositories, 14.3% of the time academic selectives, and only 7.1% of the time was it a question of public full depositories. There were minor variations in the sources used depending on the type of depository library. Figure 24 addresses this issue. Web usage at academic full, academic selective, and public full depository libraries ranges from 21.1% to 25.2% – statistically indistinguishable rates. Public selective libraries, however, used Web-based sources at a significantly lower rate – only 12.2% of the time (x²= 9.37, df=9, p<.05). Use of print sources at public full depositories and public selective depositories is around 53%, while in academic full and selective depositories, print use hovers around 40%.

One explanation for the decrease in print use at academic depository libraries, and for an approximate 20% use of Web-based sources, is offered by Dolan and Vaughan (1998), who report that academic libraries tend to have higher bandwidth connections to the Internet. This facilitates speedy access to Web pages. Since public full depository libraries are in metropolitan areas that tend to have the telecommunications infrastructure to support high bandwidth, they too use Web sources at about 20%. As smaller public selectives gain access to higher bandwidth connections, use of the Web to answer government documents questions may be expected to increase in these libraries.

Are different types of sources used in different regions of Canada? Depositories in Ontario, the Atlantic Provinces, and British Columbia make use of the Web at a statistically significant greater rate than do depository libraries in Québec and on the Prairies (x²=49.75, df=16, p <.01). Depositories in Atlantic Canada make use of the Web at a rate of 22.7% (17 out of 75 questions), those in Ontario use Web resources at a rate of 21.2% (35 out of 165 questions), while those in British Columbia employ the Web 18.9% of the time (10 out of 53 questions). These results are, for all intents and purposes, indistinguishable. In Québec, however, Web usage in depository libraries drops to 13.3% (14 out of 105 questions), while on the Prairies, use of Web resources declines yet further to 7.8% (7 out of 90 questions). Figure 25 summarizes findings about this issue.

Depository libraries in all regions of Canada use print sources at approximately double the rate that they use Web-based resources. In Atlantic Canada, print sources are used 41.3% of the time (31 out of 75 questions). Depository libraries in the Prairie Provinces use print sources at a rate of 55.6% (50 out of 90 questions), more than depository libraries in any other region. Depository libraries in Ontario also have a high rate of print use (53.3%) (88 out of 165 questions). In Québec, print sources are used 29.5% of the time (31 out of 105 questions), while in British Columbia and the North the rate of print use is 43.4% (23 out of 53 questions).

Do depositories located in larger census metropolitan areas use Web sources at a higher rate? As shown in Figure 26, there does seem to be a slight trend in this direction (x²=29.63, df=16, p <.05). In all census areas up to one million inhabitants, use of Web sources is around the 13%-16% mark, while in metropolitan areas of more than one million people, Web use increases to 22%. Depository libraries in large metropolitan areas may have access to more financial resources and sophisticated Web-based technology than their counterparts in less populous regions.

Depository libraries in the smallest census areas used OPACs at twice the rate of depository libraries in census areas with populations greater than 100,000 inhabitants. Depository libraries in these smallest census areas also used print sources much less than their counterparts in larger census metropolitan areas. The level of service at depository libraries in census areas with a population of less than 100,000 people appears to lag behind those of larger areas. This may simply reflect a lack of government documents to which patrons may be directed, or it may reflect a lack of knowledge about government documents collections by library personnel at these depositories, whose first step is often to consult their OPAC.

Were there certain types of questions for which Web sources were more popular than print sources? Figure 27 tracks sources by individual questions. In general, it is readily apparent that, for most questions, print sources were much more popular than Web-based sources.

A trend may nevertheless be discerned. There are four questions in which either Web use was greater than print use or where print and Web sources were used at approximately the same rate: the Hansard question about the Magdalen Islands; the question about a committee report on firearms legislation; the question about the Alternative Fuels Act; and the bibliographic question about the price of a government-published book. Three of these four questions deal with legislative-branch issues. Relatively high Web use (in comparison with print use) in searching for answers to these questions may indicate that staff members in depository libraries are familiar with the extensive legislatively-based information available on the Canadian Parliamentary Web site. At the same time, library personnel do not appear to be sufficiently familiar with the range of executive-branch information that is also available on the Web, since in eight out of the nine questions dealing with the executive branch they employed print sources to a greater extent than they did Web sources.

Figure 28 substantiates these observations. For the five legislative branch questions taken as a group, staff members at depository libraries used Web sources at a rate of 24.5%, while for the 10 executive branch questions, staff at depository libraries employed Web sources 14.7% of the time. Still, print sources were by far the most preferred source for both types of questions. Indeed, use of print sources was more than double that of Web sources for legislative and executive questions.

A similar result occurs when questions are categorized as data-retrieval or document-retrieval. As shown in Figure 29 below, Web sources were used to answer document-retrieval reference questions 19.2% of the time, while they were searched at a rate of 16% when data-retrieval questions were being answered. Again, the difference is slight but suggestive of the way that library personnel are using the World Wide Web to locate government information. Library staffs are more comfortable retrieving or searching documents by electronic means than they are retrieving isolated facts and statistics. One reason for this may be that they are not confident that trustworthy data-retrieval can be associated with the often ephemeral nature of Web-based documentation.

Figure 30 explores the issue of whether there is a relationship between certain types of source use and the method of question delivery. Web-based resources were used at almost equivalent rates when answering telephone and walk-in questions – 18.5% for telephone questions and 17.5% for walk-in questions. Print sources, however, were used at a greater rate for walk-in questions (55.1% of the time) than they were for telephone questions (34.6% of the time). The most troubling finding here is that "no sources" were used at a significantly higher rate for phone questions (35.2%) than they were for walk-in questions (18.2%).

As wireless telecommunications technology becomes even more ubiquitous than it is today, it is possible that a greater proportion of reference questions may be asked over the telephone. In addition, experts in demographics have noted that, in the coming decades, a greater proportion of the population will be elderly and, consequently, less mobile. Too, there are a significant number of people who have permanent or temporary difficulties with protracted physical movement. The information needs of the elderly and differently-abled are no different from members of the general population. Accordingly, depository libraries may wish to pay special attention to the data presented in Figure 30 about the discrepancy in "no source" use when questions are asked over the telephone and when they are asked in-person.

When all is said and done, library personnel consistently turn to print sources instead of Web-based resources. Reasons for this could be many. Some library staff may feel that government servers are too slow, that government search engines are ineffective, or that the necessary information is contained in Adobe Acrobat files that are either inaccessible or too large to print. Another reason for this may be the philosophy put forward by Devlin (1997) that the Internet should be chosen as an information source only if the question is unlikely to be answered elsewhere, or if other sources have been unsuccessful, or if a comprehensive search is required. Devlin's approach may be valuable for many general reference questions, but his searching strategy model may not be appropriate for government documents questions. Government information on the Web is usually reliable since it is posted by government departments and agencies themselves. Moreover, Benson (1995) suggests that, if a previously identified credible Internet source has been located, it should be consistently used as an information source. Canadian government documents are readily available on well-established and stable Web-based platforms. Library reference departments may want to consider adopting a service policy that states that, if a question seems to be a government documents question, a staff member should consult Web-based sources early on in the search.

6.7 Efficacy Rates and Source Types

It is clear that depository library personnel favour print sources by a wide margin over Internet-based Web sources. Such practice is no doubt based on long-standing habit and experience as well as the belief that complete answers may be found more readily and quickly in print sources. Are these two beliefs accurate?

Figure 31 shows complete and partially complete answers by type of source used. A good place to begin is with complete answers – perhaps the best indication of the value of individual sources. When print sources alone are used, complete answers are found at a rate of 39.9% (89 out of 223 questions). When Web sources alone are used, the complete answer rate soars to 60.7% – an approximately 50% increase in efficacy (34 out of 56 questions). Complete and partially complete answers in combination are achieved at a rate of 78.6% (44 out of 56 questions) when Web sources are used alone, while complete and partially complete answers in combination are provided 60.1% of the time when only print sources are employed (134 out of 223 questions). These differences are statistically significant (x²=10.27, df=3, p <.05).

In sum, what Figure 31 shows is a step-like progression in efficacy rates. Print is the least effective for achieving either complete or partially complete answers; the second most efficacious results are accomplished when the Web is used in combination with another source; and the best results are achieved when the Web is used as the sole source for government information retrieval.

As Table 12 demonstrates, these differences in efficacy rates between Web sources and print sources were most pronounced in public selective libraries and academic full depositories. In public selective depositories, for instance, Web sources led to complete and partially complete answers 84.2% of the time, while print sources produced the same type of answers 52.9% of the time. The difference is more muted in academic full depositories and public full depositories, although still significant. Web sources led to complete and partially complete answers at a rate of 81% in academic full depositories, while print sources provided complete and partially complete answers at a rate of 70.8%. In public full depositories, Web sources led to complete and partially complete answers 75% of the time, while print sources did so 67.7% of the time. Even though the World Wide Web as a storehouse of information and knowledge was still, in the late 1990s, in its infancy, this study offers some evidence to suggest that it has surpassed print sources in many ways as a means of retrieving complete or partially complete answers to government documents reference questions.

Table 12: Type of Answer Received by Source

Are librarians adequately prepared to find answers to reference questions using Web sources? Figure 32 provides some insight into this issue. Library personnel feel much more at ease using print sources than Web sources for questions that take nine minutes or less. The difference is striking: when print sources are used, 54.26% of the time (n=121) such sources are used for nine minutes or less. Conversely, when Web sources were used, 28.57% of the time such sources were used for nine minutes or less. This seems to indicate that library staff members, when they think that they have a good notion about where to find a piece of information, gravitate to print sources. Yet, although print sources are used 54.26% of the time (n=121), the rate of complete answers found using those print sources is 34.6%. On the other hand, despite being used only 28.57% of the time (n=16) in reference situations of nine minutes or less, Web-based sources provided complete answers at a rate of 50%. One explanation may be that Web-based resources are more efficacious than print in answering test questions of short duration. Librarians may need more training and background knowledge about the type of government information available on the Web.

Another sign of the need for additional training about Web sources is that, when staff members at depository libraries take 15 minutes or more to respond to patron questions, they are making use of Web sources at approximately twice the rate (32.14%) of print sources (17.49%). This circumstance may be an indication that they are not yet at ease with the complexities of Web-based government tools. Nonetheless, when used for 15 minutes or more, Web-based sources provided complete answers at a rate of 44.4% (n = 18), while print sources led to complete answers at a rate of 38.5% (n= 39).

Another explanation of the phenomena mentioned in the above paragraphs is that the Web qua Web is not a better tool than print, but that library staff members exploiting the Web are better skilled and more expert in comparison to their colleagues. In other words, success in finding government information on the Web is related to the searching knowledge of the Web that some library staff possess. It is interesting to note, from Table 12, that the difference between Web- and print-based resources becomes muted when comparing full depositories and is virtually non-existent with academic selective depositories, since these are the types of libraries where level of expertise might be expected to be vested in a specialist who works daily with both media and who faces a high volume of government questions. Certainly, the Web broadens the base of tools available to library staff, but staff must know how to make use of it intelligently.

There are other issues which need to be addressed before library staff make full and complete use of Web resources as a matter of natural course. The authority of Internet-based Web sources is one such major concern. Many author departments decree that print versions are the authoritative versions of texts. One government Web site, for instance, clearly states that "In the event of a discrepancy between the electronic version and a hard copy publication, the hard copy will be considered the accurate version." Another site warns that "inadvertent errors can occur for which no responsibility is accepted." In addition, librarians have reported important missing elements, especially tables and charts, from electronic copies of publications. In these circumstances, it is quite logical to expect some librarians to turn to print sources before they turn to Web sources. Authority of Web sites remains a serious issue that should be addressed. Another concern for librarians is the question of use restrictions on government sites, although some sites allow the downloading of "one copy of the materials on any single computer for your personal, non-commercial home use only."

A number of technical issues surrounding government sites may also inhibit librarians from turning to Web-based sources. For instance, the use of frames and graphics, as well as PDF formats and proprietary software such as FOLIO, are problematic, especially where public service sites have multiple functionalities. Librarians may also feel that search engines are less than adequate, given that some departments still use HARVEST and that other engines only search HTML documents and do not pick up PDF documents. Moreover, some librarians, having become accustomed to sophisticated Web-site search engines that contain such features as exact phrase, word proximity, date or database limitation, and truncation features, may find government search engines to be lacking in some of these areas.

Many documents do not contain all-important accompanying "metadata," despite the existence of basic standards for Internet publication within the government. Archiving policies are still not yet in place – the result is documents which appear and then disappear. For librarians trained in the integrity and reliable accessibility of information, this circumstance is disquieting. A final issue is that some Web sites and Web addresses are not stable, resulting in confusion for the library community and much extra work in updating electronic bookmarks.

Dolan and Vaughan (1998) present evidence to suggest that staff preparedness to help patrons with "electronic access and competent delivery of electronic government publications" is lacking because of the "absence of funding, the dearth of training programs, and the lack of time available for acquiring and passing on expertise in dealing with electronic sources of government information." The findings of the present study confirm that library personnel are not comfortable using Web-based government information sources. There are many reasons for such discomfort, as the previous paragraphs have tried to suggest. There is overall low use of Web sources in comparison with print sources; library personnel devote more time to finding answers on the Web than they do in locating answers in print sources. At the same time, this study has also shown that, when Web sources are employed, such use translates into a better reference efficacy rate, as measured by complete and partially complete answers.

Should the DSP decide to commit itself to a training and educative mission, it would be logical that Web functionality should be stressed. Nevertheless, it is important that government departments realize that improved "metadata," indexing, and archiving, as well better search engines and enhanced subject access, will go a long way in making Web-based federal information easier to find. In addition, library staff members should have a detailed knowledge of the functions of each government department and branch in order to make effective and efficient use of government Web sources. In other words, they should have a grasp of who does what in the federal government. Training courses developed by the DSP should stress this aspect of government documents reference work. Once staff members can readily identify a potential question as falling within the governmental realm through their knowledge of qui fait quoi, it becomes much easier to identify the electronic site where the desired information may be found.

7.1 The Relative Degree of Difficulty of the Questions

Previous sections of this report deal with the proxy-administered questions as a group. Attention is directed in this section to individual questions with a view to determining which were the easiest and which were the most difficult to answer. The individual characteristics of each question are examined with specific focus on the relative ease with which each can be found using Web sources. The average time spent by library staff members in answering each question is compared with the average times that the student pre-testers took to locate answers to the same questions.

Figure 33 shows the percentage of complete and partially complete answers to each of the 15 reference queries. Four questions –"crtc," "audgen," "rules," and "crime"– were supplied with complete or partially complete answers at a rate of 70% or more; three of these four are what Katz characterizes as document-retrieval questions. Nine questions had complete or partially complete answers at a rate of 30% or more; six of these nine –"audgen," "rules," "crime," "magdalen," "fisheries," and "refugee"– are also document-retrieval questions. These findings show that queries for which documents are to be retrieved appear to be easier to answer than data questions, a conclusion supported by an examination of the three questions –"books," "lyrics," and "fuels"– that show complete or partially complete success rates of less than 20%.

Two of these questions, "books" and "lyrics," clearly fall into the data-retrieval category. The "book" question asks for a specific price; "lyrics" asks for the percentage of French lyrics in Canadian-content sound recordings. The "fuels" question was classified as a document-retrieval question because it asks specifically for the text of an act, though not by name. However, this question was deliberately written in such a way as not merely to ask, for instance, "Can I get a copy of the Alternative Fuels Act?," but it requested particular data about standards connected to unconventional fuels. The wording of this question therefore tests the hypothesis that "data questions are more difficult than documents questions" insofar as another question –that dealing with the Fisheries Prices Support Act– specifically mentions its act by name and is among those nine questions for which complete and partially complete answers were found the most. Since the "fuels" question is among the three lowest for complete and partially complete answers, this lends additional credence to the fact that data questions are harder to answer than document questions, as indicated in Figure 18.

Figure 34 displays the rank descending order of questions which were completely answered. Three clear groupings of questions emerge: those for which complete answers were provided at a rate of 60%; those for which complete answers were provided at a rate of between 20% and 40%; and those questions for which complete answers were provided less than 20% of the time. By a wide margin the "audgen" and "crime" questions were those for which complete answers were provided most frequently. Again, both of these are document questions.

Figure 35 displays the rank descending order of questions which received the most no or incorrect answers. While it might be expected that Figure 35 would be the exact reverse of Figure 34, this is not the case because the different rates of partially complete answers or referrals may impact on the rates of complete and no/incorrect answers. Five questions had no/incorrect rates of 50% or more; four questions had no/incorrect rates of between 30% and 50%; six questions had no/incorrect rates of less than 30%. The two most difficult questions appear to be the "fuels" and the "firearms" question. Both of these questions are also in the group that received complete answers less than 20% of the time. These consistent results make the "fuels" and "firearms" questions those which library personnel undeniably found the most difficult to answer.

Table 13 compares the average time spent (in minutes) finding complete or partially complete answers by student pre-testers and library personnel in depository libraries. These averages do not include the time spent by library personnel when they gave referrals or provided no/incorrect answer. These averages are merely indications of tendencies.

Given this caveat, on seven of the questions (indicated with a checkmark) student pre-testers found complete or partially complete answers more than 2 minutes faster than library personnel. On six questions there was a two minute or less difference in the times spent finding such answers; for all intents and purposes, the time spent on these six questions should be considered as equivalent. On the other hand, library personnel were able to locate complete or partially complete answers significantly faster for two questions (indicated with a checkmark). Since both student pre-testers used Web-based resources to answer questions, the above data suggest that library personnel might be more efficient at finding answers were they to be better trained using Web sources.

Table 13: Average Time (in minutes) Spent Finding Answers

* all times over 20 minutes recorded as 20 minutes

7.2 Question Analysis

Question #1: Who is the Chair and other full-time members of the CRTC?

Table 14: Types of Answers Received by Type of Library for CRTC Question

One reason for asking this question was to ascertain whether staff members would think to update information contained in a directory or realize that the information in the directory was inadequate for the question asked. In the three-month period previous to this study, well-publicized personnel changes had occurred within the CRTC. These updated changes would not appear in commonly used sources such as the Canadian Almanac & Directory, but would appear on the CRTC web site http://www.crtc.gc.ca/ENG/BACKGRND/g2e.htm or in the Corpus Administrative Index. Credit for a partially complete answer was given to depository libraries whose staff used a single source such as the Canadian Almanac; credit for a complete answer was given for the ability to provide the most recent information. One striking aspect of the answers received was that many library staff, especially at public selectives, did not update the answer.

Question #2: I want to order a copy of Aboriginal Self-Government by Jill Wherrett, published in 1996. I'm sure it's a government document, and I specifically want to know how much it costs and any ordering instructions.

Table 15: Types of Answers Received by Type of Library for Book Question

On the surface this is a difficult question, but a complete answer to this question could be easily found by searching the Depository Services Program Web site at the following address: http://dsp-psd.communication.gc.ca/search_form-e.html. Here, a search engine is available where a search by author can be performed. Three references to a book entitled Aboriginal Self-Government appear; two of these refer to books for which Wherrett is a co-author with Jane Allain. The book for which Wherrett is sole author is the book asked about in this question. Clicking on this title shows that the price of the book is $6.50. A noteworthy aspect of this question is the high number of referrals made to various types of local bookstores. A number of full depositories had copies of this volume, and suggested that the patron come in to view the book, but these libraries could not provide the price.

Table 16: Types of Answers Received by Type of Library for Barley Question

A complete answer for this question could be found in a number of locations, although, again, a relatively easy place to locate it is at the Web site of the Canadian Wheat Board, available at http://www.cwb.ca. Once here, click under Payments. Payments are listed for specified years, and are given either in tonnes or in bushels. In print, the answer is available, in tonnes, from a publication entitled Grain Trade of Canada. The introduction to this book states how many bushels there are in a tonne. On this question, an answer was judged to be complete if any price in tonnes was provided for a barley product. Only two library staff members referred to the bushel/tonnes conversion from print sources. In addition, only two staff members consulted the Canadian Wheat Board Web site. Just as many no/incorrect answers (15) were received to this question as the sum of complete (12) and partially complete answers (3).

Question #4: I'd like to know how many new Canadian-content sound recordings (albums, tapes, CD's) released during 1990-1994 have French lyrics?

Table 17: Types of Answers Received by Type of Library for Lyrics Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	2	0	3	2	7
Academic select	2	0	0	1	3
Public full	4	0	0	1	5
Public selective	9	1	6	1	17
Grand Total	17	1	9	5	32

Answers to this question could be found using Statistics Canada sources, either print or electronic. In print, a publication entitled Sound Recordings (#87-202) could be used to locate the required information. The Statistics Canada Web site http://www.statcan.ca also contained the answer. Here, choose the subsection Canadian Statistics, then People, then Culture, Leisure, then Profile of the Sound Recording Industry. Many library staff members were unwilling even to attempt answering this question; one proxy was referred to a local record store. On six of the 12 occasions when this question was asked at a full depository, proxies received an incorrect answer or no answer. More than half of the answers (17) to this question were no/incorrect.

Question #5: I'd like to get the text of the act that requires crown corporations to power their motor vehicles with fuels that do not harm the environment. How many of their vehicles have to use these non-conventional fuels in fiscal 1998?

Table 18: Types of Answers Received by Type of Library for Fuels Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	8	0	2	4	14
Academic select	2	0	1	0	3
Public full	2	0	0	1	3
Public selective	8	0	3	1	12
Grand Total	20	0	6	6	32

This question attempted to test whether library personnel were aware of the search engine available at the Web site of the Justice Department. The question did not specifically ask for a statute by name in order that the search engine facilities might be used. Full texts of Canadian laws are available at http://canada.justice.gc.ca. Click the search icon. Then enter the following search string in the query box: ["motor vehicle*" and "crown corporation*" and fuel*]. All these terms were specifically stated in the text of the question. The very first hit is the Alternative Fuels Act, in which the complete answer about how many vehicles owned by crown corporations must be powered by non-conventional fuels can be found. The answer is also available in print from the Statutes of Canada 1995. Of interest here is a not untypical response from a staff member at an academic full depository: "I am unable to answer your question. It would take a lot of digging and research to find the stats and the act itself…." Again, more than half the responses received (20) were judged to fall in the no/incorrect category.

Question #6: There was a parliamentary sub-committee on the draft regulations on firearms that submitted a report to the House of Commons in 1997. I'd like to see a copy of this report.

Table 19: Types of Answers Received by Type of Library for Firearms Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	5	1	0	6	12
Academic select	8	0	1	0	9
Public full	1	1	0	0	2
Public selective	6	1	3	0	10
Grand Total	20	3	4	6	33

This question tested whether library staff members could find committee reports. A relatively simple way to locate this report was through Web-based resources. Go to the main parliamentary home page at http://www.parl.gc.ca/36/main-e.html. Click on the heading entitled Site Map. From there, scroll down to the sub-section labelled Committees. Pick the committees for the House of Commons, then select Reports. Then scroll through the various committees until the Standing Committee on Justice and Legal Affairs. The final report is available at http://www.parl.gc.ca/36/1/parlbus/commbus/house/juri/reports/jurirp04-e.htm. One surprising finding here was that some library staff members went immediately to CBCA – the Canadian Business and Current Affairs Database. Another worrisome finding was the performance of full depository libraries – on six of the 14 occasions that this question was asked at full depositories, no answer or an incorrect answer was provided.

Question #7: I'd like to know if the Auditor-General said something in the 1992 annual report about forest management practices of natives, specifically about the good job done by the Stuart Trembleur Lake Band.

Table 20: Types of Answers Received by Type of Library for Audgen Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	2	0	0	7	9
Academic select	0	0	0	4	4
Public full	1	0	0	2	3
Public selective	2	1	5	9	17
Grand Total	5	1	5	22	33

This was one of the two questions for which the greatest number of complete answers was received. Many libraries had print copies of the 1992 Auditor General's report, and library staff showed patrons how to use it. The answer is also available electronically at the Web site of the Auditor-General: http://www.oag-bvg.gc.ca/domino/reports.nsf/html/92menu_e.html. The answer is in Chapter 15, subsection 60, where the Stuart Trembleur Band is praised for sound forest investment and management. Also noteworthy is the fact that very few no/incorrect answers were received to this question.

Question #8: I'd like to see a bill that was introduced into the House of Commons this past fall. It has to do with the profits convicted criminals might make if they were to publish books about their crimes.

Table 21: Types of Answers Received by Type of Library for Crime Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	3	1	0	7	11
Academic select	2	0	0	5	7
Public full	1	0	0	3	4
Public selective	4	0	0	7	11
Grand Total	10	1	0	22	33

As with the previous question, 22 complete answers were received for this question. A caveat, however, must be attached to these impressive results. Even though the question specified that the bill was introduced in the fall of 1997, an answer was coded as being complete if the library staff member located any of the three versions of the bill introduced in the past three years. This private member's bill, entitled "An Act to amend the Criminal Code and the Copyright Act (profit from authorship respecting a crime)," received a great deal of media coverage during the autumn months of 1997. It can be found through the Parliament Web page of the federal government. Within the 36^th Parliament, choose the sub-section headed Private Member's Bills, then scroll down to the appropriate bill, numbered C-220. The final address is http://www.parl.gc.ca/cgi-bin/36/pb_prb.pl?e. When using this Web service, a bill number is not required. Some library staff members, however, told patrons that bill numbers were required to locate the required information. Other librarians unsuccessfully made use of such Internet search engines as Infoseek by typing in keywords "crime" and "profits."

Question #9: I'm doing a class project about the Magdalen Islands, and there was talk about closing the marine radio station there. I'd like to know if anything was said in the House of Commons about this topic in the last year, and if anything has been decided about its fate.

Table 22: Types of Answers Received by Type of Library for Magdalen Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	4	4	0	2	10
Academic select	2	1	0	2	5
Public full	2	4	0	0	6
Public selective	7	1	3	1	12
Grand Total	15	10	3	5	33

This question deals with Hansard, the official record of debates in the House of Commons. While the subject matter of the question might seem esoteric, the question is intended to test ability to identify and use Hansard. The basis of this question would be no different were a question to be asked about what, for instance, Alexa McDonough or Preston Manning said about any political issue. Go to the House of Commons debates sections of the Parliamentary Home Page at http://www.parl.gc.ca/36/1/parlbus/chambus/house/debates/indexe/homepage.htm. An alphabetical index of subjects and the names of members is located here. Since information about the Magdalen Islands is required, scroll down to the M=s. Click on the M, then scroll down until the subject heading for "Magdalen Islands" is reached. Click on any of three documents until relevant information is found. The answer is that the federal government is "not closing the station but rather it will be operated from Rivière-au-Renard." A decision is still pending about whether to relocate workers from Cap-aux-Meules. As suggested in Table 21, very few complete answers were received to this question. Many library staff consulted the compendium publication Ottawa Letter. Many others simply took patrons over to the print issues of Hansard for which no indexes were available and suggested that they look through the accumulated issues themselves. One exasperated referral was made to the local Member of Parliament.

Question #10: I'd like to know the complete set of rules that govern Question Period in the House of Commons.

Table 23: Types of Answers Received by Type of Library for Rules Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	0	4	0	1	5
Academic select	0	0	0	1	1
Public full	1	3	0	2	6
Public selective	6	8	0	7	21
Grand Total	7	15	0	11	33

The answer to this question can, again, be found from the main Parliamentary page. Go to http://www.parl.gc.ca. Choose Reference Material from the available subjects on this first page. Then scroll down to Reference Works – Procedural. The correct answer is contained in the Standing Orders of the House of Commons. As well, a print version of the Standing Orders exists. One interesting aspect about this question is that only two library staff members chose to use a Web-based source. While 11 depository libraries completely answered this question, those whose answers were recorded as partially complete showed patrons either the Précis of Procedure or a general work about the functioning of Parliament. Of more concern is that a number of library staff members showed patrons reference works dealing with procedures in the Congress of the United States. When questions were being developed, this question was thought to be one of the easiest, since it only involves directing the patron to the Standing Orders. It is disquieting, therefore, to find that two-thirds of the library staff approached for this question failed to identify this major reference tool.

Question #11: I want to know if there is any official document about the possibility of immigrating to Canada as a refugee claimant because of persecution based on gender.

Table 24: Types of Answers Received by Type of Library for Refugee Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	3	0	1	3	7
Academic select	4	0	1	1	6
Public full	2	1	1	0	4
Public selective	7	6	2	0	15
Grand Total	16	7	5	4	32

The results received for this question are an indication of the difficulty government documents reference staff experience when dealing with what appears to them to be a judicial question. Fully half of the responses fall into the no/incorrect category, while only four of the answers were categorized as being complete. This is especially disturbing, given the potential importance of the subject matter of the question to a patron. One common location to find the answer in print is as an appendix attached to a report by Margaret Young entitled Gender-related Refugee Claims (1994), published by the Laws & Government Division of the Library of Parliament. The required information is also accessible through the Web site of the Immigration and Refugee Board of Canada, which is available at http://www.cisr.gc.ca. From the home page of this site, choose the subject heading Legal References. The very first screen of this hyperlink contains a section entitled Chairperson's Guidelines – Women Refugee Claimants Fearing Gender-Related Persecution. Some library staff members consulted only ready reference sources such as the Canada Yearbook. Others directed proxies to informational leaflets about immigrating to Canada or the Self-Counsel series on the same topic.

Question #12: Someone I know is looking for work hauling garbage. Would there be any specific opportunities to put in bids for contracts in this field with the federal government?

Table 25: Types of Answers Received by Type of Library for Garbage Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	0	0	1	4	5
Academic select	4	0	0	1	5
Public full	2	0	0	2	4
Public selective	3	2	8	5	18
Grand Total	9	2	9	12	32

Like the question dealing with the Magdalen Islands, the subject matter of this question may at first appear to be obscure. However, a patron could ask for bidding opportunities connected with any field of endeavour, and the answer could be found in the same location. This is the type of practical question that may be of great financial importance to certain patrons. Probably the best print source is the weekly or bi-weekly Government Business Opportunities, although there is no index in this publication. An electronic source is the Public Works Canada Web site at http://contractscanada.gc.ca/en/index.html. Then go to Database of Current Government Bidding Opportunities at http://contractscanada.gc.ca/en/tender-e.htm. This is the MERX system at http://www.merx.cebra.com. Type in "garbage" under Opportunity Search. This system was monitored throughout the time period for this study; there were many contracts for hauling garbage available. A substantial number of library staff members gave only the vaguest possible answers to this question. Some referred the proxy to "government offices that dealt with this field"; others told the proxy "to go and see the minister who deals with garbage disposal"; still others referred the proxy to local municipal authorities.

Question #13: My mother's birthday is coming soon, and I want to order a color enlargement of an aerial photograph of the lake where my parents have their summer house as her present. Could I have a price list for the enlargements, and information about what I need to do to order such a photograph?

Table 26: Types of Answers Received by Type of Library for Photo Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	0	0	7	0	7
Academic select	0	0	2	1	3
Public full	1	1	2	1	5
Public selective	4	2	8	3	17
Grand Total	5	3	19	5	32

One reason that this question was selected for inclusion in the study was that it tests the knowledge of library personnel about government services without specifically mentioning that aerial mapping and aerial photography can be provided by the federal government. The most complete source is available on the World Wide Web. Go to the Geomatics Canada site at http://www.geocan.nrcan.gc.ca. Then, under Thematic Mapping, there is a link to the National air photo library at http://airphotos.NRCan.gc.ca/main.html. This site is searchable by key words; choose prices, enlargement, or ordering. Price lists for enlargements can be found at http://airphotos.NRCan.gc.ca/prices.html. Many provinces also have departments that provide aerial photography. In addition, by using the Yahoo Canada Directory Web site, available at http://www.yahoo.ca, the National Air Photo Library is the first returned hit if the search term "aerial photo" is entered. One significant aspect about this question is the large number of referrals to local photography shops. To be sure, some local photographers would be able to direct patrons to the proper government agency; nevertheless, the inability of some library personnel to identify a government service is cause for concern.

Question #14: Can you help me find any regulations or enabling statutes associated with the Fisheries Prices Support Act?

Table 27: Types of Answers Received by Type of Library for Fisheries Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	5	0	1	4	10
Academic select	3	0	0	1	4
Public full	0	0	0	2	2
Public selective	9	0	3	4	16
Grand Total	17	0	4	11	32

Unlike Question #5 above, this question specifically mentions a statute by name. But answers to both questions may be located in the same place, namely, the Justice Canada Web site available at http://canada.justice.gc.ca. From this site, choose Laws, then Text Versions of Statutes and Associated Regulations for Download. Statutes and the regulations that pertain to them are listed in alphabetical order. Click on the letter F, and then scroll down to the Fisheries Prices Support Act. There are three entries for regulations: Canned Mackerel Support Order; Frozen and Cured Herring Price Support Order; and the Small and Extra Small Heavy Salted Dried Cod Price Support Order. These regulations were last updated in 1994. More than half the answers were received fell into the category no/incorrect. Some staff members at full depositories tried to use the 1985 version of Revised Statutes of Canada, clearly an outdated tool. Some library personnel employed the Canada Gazette Part II: Consolidated Index of Statutory Instruments to locate the list of the regulations, but did not search for the texts of the regulations. In an example of persistent and high-quality service, one library staff member spent slightly more than an hour with a patron in a successful attempt to track down these regulations.

Question #15: Does any government department put out any newsletters or bulletins about business opportunities in Africa? If so, I'd like a copy of the latest one.

Table 28: Types of Answers Received by Type of Library for Africa Question

	No/incorrect	Partially complete	Referral	Complete	Grand Total
Academic full	1	0	1	2	4
Academic select	3	1	0	0	4
Public full	1	0	2	1	4
Public selective	8	2	7	3	20
Grand Total	13	3	10	6	32

This question tests the ability of library personnel to find periodicals published by the government. Again, this is the type of question that may have immediate practical consequences for some patrons. The answer could readily be found using the Web site of the Department of Foreign Affairs and International Trade at http://www.dfait-maeci.gc.ca. From here, the library staff member has a number of equally valuable options. After glancing at the main page, the following sub-headings could be linked to: News Releases, Statements, and Publications, under the section The Department; Market Information, under the section Trade; Africa & Middle East, under the section The World. Under News Releases, Statements, and Publications, go to the section labelled Publications; then choose Trade. Under this heading, one can find a publication called "Africa & Middle East Bulletin," which briefly summarizes the potential that African countries have for Canadian businesses. Clicking on Market Information leads to a page entitled Market Reports: Information by Region and Sector. Choose the African sector; this leads to an alphabetical listing of countries. Choosing a country leads to detailed information about trade, exploration, and export opportunities. Although some information is password protected, any Canadian citizen is issued a password after completing a basic informational form.

Under "The World" heading, choosing Africa leads to a cornucopia of business information about this region. For instance, there is a publication entitled "The African Development Bank Group: A Guide to Business Opportunities for Canadians." Only six complete answers were provided to this question, while 13 no/incorrect answers and 10 referrals were registered. Some library personnel suggested calling relevant African embassies in Ottawa; others gave out publications listing overseas jobs, most of which were in Asia or South America. One suggested calling UNICEF; another typed in the keywords "Africa" and "business" into an online catalogue and told the proxy to browse the shelves for relevant books. Still another suggested that the proxy consult an Ethiopian newspaper carried by the library.

---

Conclusions and Recommendations

This study was undertaken to discover: (i) the degree of accuracy of government reference service in Canadian academic and public libraries participating in the Depository Services Program (DSP), as measured by the number of correct answers supplied to test questions asked by proxies; (ii) the extent to which staff members make use of electronic resources, in particular federal government Web sites, in answering the questions; and (iii) which categories of government reference queries do depository reference personnel find most difficult to answer.

In regard to external institutional variables, findings show that, overall, depository library staff members provided complete answers to questions 29.3% of the time. When complete and partially complete answers are taken together (reflecting a conservative reference philosophy as outlined in Section 3.9) the success rate climbs to 42.4%. No/incorrect answers were given at a rate of 37.6% while a full one-fifth of the 488 questions were referred to, mostly to non-specific external sources (Figure 7).

The rate of 29.3% for complete answers is disappointing. It may be indicative of the difficulties commonly associated with the quantity, variety, and complexity of collections of official publications. It may indicate that library staff members are not at ease in finding their way through the maze of documents, whether print or electronic. It may also reflect the increasing pressures on libraries suffering from budget reductions and the growing demands of the new technologies, as reported in Dolan and Vaughan (1998).

There were notable differences among the four types of depositories in providing complete answers. Academic full depositories achieved the highest rate of success, followed by public full depositories. Academic and public selective libraries did less well. When complete and partially complete answers are taken together, academic and public full depositories performed at an almost identical rate, as did both types of selective depositories (Figure 8). Higher success rates among full depositories are no doubt attributable to their higher staffing levels, more specialized service, and access to the full range of DSP publications.

Service levels differed as well according to region and census metropolitan area. Among the five geographical regions, Ontario performed best, followed by British Columbia and the Atlantic Provinces, then by the Prairie Provinces and Québec (Figure 9). Levels of documents service varied according to type of library in the regions. In Québec, the Prairies, and British Columbia, academic full depositories offered the best service. In Atlantic Canada and Ontario, however, similar degrees of service were provided by all four types of depositories, with the exception of academic selectives in the Atlantic Provinces. With regard to census metropolitan areas, it was found that the lowest number of complete and partially complete answers to reference queries was provided in cities with a population of fewer than 100,000 inhabitants. Census areas with populations of over one million or between one quarter and one half million inhabitants offered the best opportunities for complete or partially complete answers to government-related questions (Figure 10). As for the days of the week on which proxy questions were asked, it was found that on most days service was remarkably similar. The rate for complete and partially complete answers rose on Tuesday and declined on Sunday. These differences may be the result of staffing levels that fluctuate over the week (Figure 11).

Internal institutional variables were an important part of the study reported on here. Full depositories with separate reference desks or areas for official publications provided more complete or partially complete answers than those without such areas. This tendency was most pronounced in public full depositories where the difference in success rates was almost 17%, although the 10% variation in academic full depositories is also noteworthy (Table 7). This finding supports the traditional notion that the size and complexity of government documents collections require special attention and expertise on the part of librarians.

The degree of busyness at reference desks and whether it had an impact on the quality of answers was examined. Academic full depositories provided complete and partially complete answers at the same rate of about 50% whether they were busy or not. As for time spent in reference encounters, it was found that as the amount of time spent with proxies increased, the number of complete and partially complete answers went up significantly. With referrals, the opposite tendency was observed; the referral rate was lowest where time spent was greatest (Figure 15). Results suggest that, with enough time and opportunity, library staff members are able to achieve a very high rate of complete and partially complete answers.

A greater number of complete and partially complete answers was received when reference questions were asked in person rather than over the telephone, especially in full depositories; more referrals were given to telephone questions. Selective depositories, on the other hand, were able to supply answers to telephone and walk-in questions at an equal or almost equal rate (Table 9).

One of the most interesting findings in this study shows that legislative questions were answered completely and partially completely at a significantly higher rate than were queries dealing with the executive branch of the federal government. Moreover, executive questions were referred at a substantially higher rate. This was true for all libraries in the study, but the difference was most striking in academic and public full depositories (Table 11). One explanation for this may be that legislative questions are more obvious, and that sources for answers to them more limited in number.

Referrals comprised one-fifth of responses to proxy-administered questions. Half were to government departments, 36% to other libraries, and 14% to external non-governmental agencies or commercial establishments (Figure 19). Most referrals made to government departments were made by public selective libraries, but both academic and pubic full depositories made referrals 16.3% of the time. No/incorrect answers were given at a rate of about 38%. The most common explanation for this rate was that the staff members did not know how to find the needed information. Inaccurate responses made up 21% of no/incorrect answers and 10% resulted from the disinclination of library staff members to answer proxy-administered questions. Finally, 27% of the time proxies were told to come into the library, telephone at a later time, or return at a more convenient time (Figure 20).

No/incorrect answers were provided at approximately the same rate (34%-36%) among academic and public full and public selective libraries, and at a significantly higher rate at academic selective libraries. Referrals were made by academic full, academic selective, and public full depositories at an approximately equal rate ranging from 9% to 15%. For public selective libraries, the rate was almost 30% (Figure 21). Despite the widespread availability of Web resources in Canadian depository libraries, findings show that print materials were by far the largest single source used to answer questions in this study. The Web alone was used at a rate of only 11.5% and the Web in combination with another source a mere 5.5% of the time (Figure 23). Web sources were used at a greater rate in Ontario, the Atlantic Provinces, and British Columbia, than in Québec and the Prairie Provinces (Figure 25). There is a slight increase in the use of Web sources in the largest metropolitan census areas (Figure 26)

For most questions, print sources were preferred much more often than those on the Web (Figure 28). Web use was greater in the search for three legislative-branch questions and one bibliographic question. It is reasonable to surmise that library staff are more familiar with legislative information on the Parliamentary and DSP sites and that they have insufficient knowledge about the extensive range of executive branch information on the Internet. There is a slight difference in the degree to which the Web is used to search for document-retrieval questions (19.2%) and data-retrieval questions (16%), suggesting that reference staff members are more at ease retrieving documents electronically than they are searching for isolated facts and statistics. There may be less confidence placed in electronic data, given its often ephemeral nature (Figure 29).

Rates were essentially the same when it came to using Web sources for telephone and walk-in questions, although print sources far surpassed electronic means for walk-in queries as opposed to telephone requests. More troubling is the finding that "no sources" were used at a significantly higher rate for phone questions than they were for walk-ins (Figure 30).

Results demonstrate that reference staff favoured print sources by a wide margin, yet when print alone was used, complete answers to the test questions were found only 39.9% of the time. When Web sources alone were used, the complete answer rate was 60.7% (Figure 31). Data also suggest that Web-based information sources are more effective than print in searches of short duration.

Based on the findings of this study, depository libraries are not turning to the World Wide Web at a great rate to aid patrons find government documents. Print sources are the overwhelming first choice of library staff members when faced with government document questions. Yet print sources are not nearly so efficacious at finding complete answers as are Web-based informational sources. As indicated by the data presented in Figure 31, when print sources alone were used, complete answers were found at a rate of 39.9%. When Web-based sources were used, however, complete answers were found at a rate of 60.7%. As indicated earlier, one reason for this may be that the material provided to depository libraries by the DSP is poorly indexed, lacks consistency, and comes with little training support.

This study also found varying rates of complete answer provision between full and selective depository libraries, among different regions of the country, and among varying sizes of metropolitan census areas. Accordingly, one advantage of Web-based government information sources is that they can level the playing field between types of libraries and between different regions of the country. All libraries, whether full or selective or neither, have access to the same body of information and documents provided by the Canadian federal government.

The federal government is moving to implement a plan in which the preferred delivery platform for government information will henceforward be electronically-based through the World Wide Web. One consequence of this will be that the distinction between a full depository library and a selective depository library will disappear. Indeed, all computer-owning individuals will have the same access to federal government information as the largest library in the country. However, not everyone will have access to a computer and the World Wide Web. Too, those individuals who do have such access may not be very proficient at finding their way around this new information medium in their search for specific government documents and data. Libraries therefore still have an important role to play as intermediaries between government information and the general public, but if and only if they are able to provide superior government documents reference service. And if they are to offer a new and superior level of service, one requirement must be increased attention to training staff members in efficiently locating government documents and data. The DSP can play a central role in the training of library staff members to ensure that staff members are knowledgeable about government functions and are able to efficiently and effectively retrieve government documents. A new manual or text will go a long way to assist any training endeavours.

To make use of these Canadian government Web resources effectively, it is vital that library staff members are fully aware of the structures and functions of both the legislative and executive branches of government. Staff members need to know what programs are available and who is responsible for which program in the federal government. In short, library staff should be knowledgeable about who does what and how things work. The DSP might undertake this type of training of library staff as a central part of its mission. The DSP might also consider entering into working agreements with provincial and municipal levels of government in order to develop and implement city- and province-specific learning programs about all available governmental services for a particular census metropolitan area. In this way, the DSP could expand its mission to include "who does what" at the non-federal level.

The DSP might also do more to facilitate electronic access to federal documents. Certainly, the DSP has made great strides in this direction through its own Web site. Recently, it has also worked together with Anita Cannon, Reference/Public Service Librarian at Mount Allison University, Sackville, New Brunswick, to create subject-specific abstracts about various government departments. Cannon's web site contains valuable information and links. It is available at http://library.uwaterloo.ca/discipline/Government/CanGuide/Federal.html. However, Cannon's site serves only as a broadly-descriptive roadmap. The DSP might wish to stress improved "metadata," indexing, and archiving of its Web-based information, as well as improved search engines and enhanced subject access. A carefully conceived and complete blueprint for improving access to government documents is provided in the Whitepaper on Government Information in the Electronic Environment, published by the Ad Hoc Committee on the Internet, Government Documents Roundtable (GODORT), of the American Library Association. This report is available at http://www.lib.berkeley.edu/GODORT/whiteppr.html. Particularly relevant, in the context of the present study, are the extensive sections on "Preserving and Archiving Electronic Government Information" and "Education Issues."

In discussing preservation and archiving, GODORT makes a number of salient recommendations that the DSP may wish to consider. First, there is the question of standards and data loss. Because the hardware on which information is stored and the software used to access it are "constantly upgraded and superseded," it may be difficult in the future to work with "older portable electronic sources." Government agencies should therefore be "responsible for maintaining permanent access to their entire historical electronic publication record." Moreover, because of difficulties experienced with the "file integrity" and "life expectancy" of different types of digital storage formats, governments wishing to maintain digitally-stored information on a long-term basis may want to consider codifying standards of "data readability, data retrievability, and data intelligibility." Another issue –called authentification and data-fixing– is the provision of "consistent access" to digital information "through citation and retrieval over time." It is important "to refer to a source over time and assume with reliability that it has the same content that it did when originally cited." Sophisticated digital signatures may be one answer to preventing modification, transformation, and censorship of original government data.

Whereas in previous decades, the DSP provided access to government documents by physically distributing these documents to depository libraries, now, at the dawn of a new century characterized by increasingly powerful and sophisticated electronic information technologies, documents are instantaneously distributed through electronic means. But, without exhaustive indexing, reliable "metadata," and coherent archiving standards, these documents are no more accessible than if they were simply stored in a warehouse. In a very real sense, then, the DSP still needs to provide access to these already-distributed documents. This access is best provided through superior indexing, enhanced subject access, and consistent data retrievability of authentificated and "fixed-in-time" government documents. The mission of the DSP will thus evolve to meet the changing circumstances and needs of the 21^st century, although it will still adhere to its historic mission of ensuring egalitarian access to federal government documents.

What should be the future role of depository libraries? Libraries may find it natural to become the preferred location for training programs geared toward making the library the central location for dissemination of various types of government documentation and information. The DSP may want to become the prime agent for achieving such training and education through frequent regionally-based in-person seminars or through regularly-updated distance-learning modules that library personnel would complete on a mandated-timetable basis. Training would cover "who does what" programs as well as sessions about DSP electronic services, including enhanced subject access, improved "metadata," better search engines, and archiving initiatives.

---

Bibliography

Benson, Allen C. (1995). The complete Internet companion for librarians. New York: Neal-Shuman Publishers, Inc.

Canada. Transport Canada. Administrative Services. (1986). You asked us: Typical questions and answers handled by the inquiry desk of the Transport Canada Library and Information Centre. Ottawa. [TP7555]

Childers, Thomas. (1987). The quality of reference: Still moot after 20 years. Journal of Academic Librarianship, 13, 73-74.

Devlin, Brendan. (1997). Conceptual models for network literacy. The Electronic Library, 15, 363-368.

Dolan, Elizabeth, & Vaughan, Liwen Q. (1998). Electronic access to federal government documents: How prepared are the depository libraries? Ottawa: Canada Communications Group.

Durrance, Joan C. (1989). Reference success: Does the 55 percent rule tell the whole story? Library Journal, 114 (6), 31-36.

Elzy, Cheryl, Nourie, Alan, Lancaster, F. W., & Joseph, Kurt M. (1991). Evaluating reference service in a large academic library. College & Research Libraries, 52, 454-465.

Gers, Ralph, & Seward, Lillie. (1985). Improving reference performance. Library Journal, 110(17), 32-35.

Harris, Roma M., & Marshall, Victoria. (1998). Reorganizing Canadian libraries: A giant step back from the front. Library Trends, 46, 564-580.

Hernon, Peter, & McClure, Charles R. (1986). Unobtrusive reference testing: The 55 percent rule. Library Journal, 111(6), 37-41.

Hernon, Peter, & McClure, Charles R. (1987). Unobtrusive testing and library reference services. Norwood, NJ: Ablex Publishing Company.

Hults, Patricia. (1992). Reference evaluation: An overview. The Reference Librarian, 38, 141-150.

Jardine, Carolyn W. (1995). Maybe the 55 percent rule doesn't tell the whole story: A user-satisfaction survey. College & Research Libraries, 56, 477-485.

Katz, William. (1982). Introduction to reference work, 5^th ed. New York: McGraw-Hill.

Katz, William. (1996). Introduction to reference work, Volume 1: Basic information services, 7^th ed.

New York: The McGraw-Hill Companies, Inc.

Lancaster, F. W. (1977). The measurement and evaluation of library services. Washington, DC: Information Resources Press.

McClure, Charles R., & Hernon, Peter. (1983). Improving the quality of reference service for government publications. Chicago: American Library Association.

McIlroy, Anne. (1998, February 12). One-third of stores selling tobacco to minors. The Globe and Mail, A1, A10.

Murfin, Marjorie E. (1995). Evaluation of reference service by user report of success. The Reference Librarian, 49/50, 229-241.

Parker, June D. (1996). Evaluating documents reference service and the implications for improvement. Journal of Government Information, 23, 49-70.

Richardson, John V., Jr. (1998). Question Master: An evaluation of a Web-based decision-support system for use in reference environments. College & Research Libraries, 59, 29-37.

Steinhauer, Jennifer. (1998, February 4). The undercover shoppers: Posing as customers, paid agents grade the stores. New York Times, C1, C23 [National].

Tyckoson, David A. (1992). Wrong questions, wrong answers: Behavioral vs. factual evaluation of reference service. The Reference Librarian, 38, 151-173.

Whitlatch, Jo Bell. (1989). Unobtrusive studies and the quality of academic library reference services. College & Research Libraries, 50, 181-194.

Note: All Web addresses mentioned in this report were valid (or operational) during the period that this study was conducted.

1 "Introducing the DSP" http://dsp-psd.communication.gc.ca/dsp-psd/Info/profile-e.html

2 Canadian Libraries that are nominated and approved by a Committee consisting of representatives of the National Library of Canada and the Depository Services Program are granted full depository status which is designated as either English, French, or bilingual, depending on the library's clientele.

3 Selective depository status is granded to Canadian public libraries and libraries of educational institutions which are open to the general public at least 20 hours per week and have at least one full-time employee. Twelve selective depositories are not libraries.

4 Public full depository and public selective depository figures are reported together for Québec, British Columbia, and the Prairies because, in each region, there is only one public full depository library. Identification of this library would therefore be possible. Anonymity is preserved by combining results for both types of public depositories.

5 Proxies could not answer with certainty about this issue 3% (15 questions) of the time.

6 The chi-square test mentioned in this paragraph and the next two chi-square tests (all three involving various types of sources) are calculated using the five major types of sources used by depository libraries: print; Web alone; Web combined with an another source; OPAC; and no sources. Row totals and column totals thus add up to 465, instead of 488 (total questions). CD-ROM sources (18) and microform sources (5) were not included because of cell size criteria.