Research on 'Quality' in Online Experiences for Museum Users

Measuring Quality of Existing Online Museum Products in the Virtual Museum Portal

Quantitative Analysis

Tracking Online Visitors

Tracking on-line experiences has become a rigorous process for gathering statistics on visitor use. When someone visits a page on a Web site, a request is made to the Web site's host computer to send electronic information about the visitor's computer. Those requests are tracked as ‘hits’. Basically, a hit registers each time a visitor clicks on part of the site. A hit also registers for each graphic or link on a page. Therefore if a visitor merely looks at a page with four graphics on it, that visitor registers as four hits. These statistics are stored in a log file on the Web site server and may be viewed on screen or printed out. In addition to basic log file tracking, various data collection software is also available (such as WebTrends^TM, Summary,^TM Webalizer^TM) to produce more detailed reports, charts, and tables with visitor statistics.

Comparison of Statistical Data of VMC Products

Visits & Visitors

Virtual Exhibits with globally appealing topics have the highest number of Visits. Christmas Traditions, which peaks seasonally in November and December; ranks highest in Visits and Visitors (1^st), and Panoramas about North American landscapes ranks next highest (2^nd). Butterflies North and South about an exquisite non-threatening insect also has a high Visit ranking (3^rd), as does Haida Spirits of the Sea about indigenous peoples (4^th).

Visits / Page Views

Visits / Page Views shows more consistency than number of Visits and Visitors to a Virtual Exhibit or Game across the Engagement Factor sample. The three highest Engagement Factor sites are evident - Butterflies North and South(highest in Page Views) and (second highest in Page Views).

Duration

Of the three Engagement Factor statistics that can be obtained through log files, statistics show the most variability in Duration. Duration of time spent online seems more important to determine the quality of the Web site; the longer the user stays the more ‘engaged’ they are likely to be.

(Haida ranks 7^th in Duration) compared with Christmas Traditions (ranked 19^th for Duration, and particularly low at Christmas time when there are the most Visits and Visitors), and Panoramas (ranked 23^rd in Duration). Butterflies North and South had more Visits / Visitors in March when it was Featured, and fewer in November.

Interestingly, users tend to spend more time on the following Virtual Exhibits in language versions other than English (particularly those in French, Spanish, and Portuguese). Haida Spirits of the Sea; Explore Herschel Island!; Yukon Photographers; The Astonishing World of Musical Instruments; Christmas Traditions; and Panoramas (yet less time in Perspectives). The language differences may be because of modem speeds, search engines used, and the amount of related online content in different languages. Users tend to spend equal amounts of time in the different language versions of the following sites: Butterflies North and South ; Staying in Tune; and Hockey. The topics of these sites – butterflies, music, and a popular Canadian sport – may be easier to access across languages.

Note: The Engagement Factor ranking is not an actual ‘score’ but a ranking of the scores. This is also the case for Statistics, therefore, the lower the ranking the higher the Engagement Factor.

Web Site Traffic

The following four charts show Web site traffic of twelve VMC products collected during the year 2002.

Understanding Engagement Factor

Haida Spirits of the Sea [EF 3^rd] / Butterflies North and South [EF 9^th]

Haida Spirits of the Sea and Butterflies North and South are both deserving of their high Engagement Factor ranking. They contain key elements in design and contents that would seem museologically to provide for a more engaging online experience (e.g., compelling narratives, depth of information, good information strategies, and clear navigation and organization).

Butterflies North and South has considerable depth of information and meaningful content. Content for Haida Spirits of the Sea was given freely, from the heart and soul of the Haida. Although catering to different audiences, both launch dates and approaches are more educational than edutainment sites and continue to be very successful VMC products.

Yukon Photographers [EF 10^th] / Explore Herschel Island! [EF 12^th]

Both of these sites were created by the same collaborative group in the Yukon and launched on the VMC site in 2001. Both are content-driven and informative, but more like traditional exhibits than sites that take advantage of the potential of online experiences for users (the Web still seems to be driven by content rather than ‘edutainment’ or ‘entertainment’).

There are elements in the design of both Virtual Exhibits that decrease the potential for user engagement, such as linearity, limited possibilities for interactivity and self-directed exploration, little directed content in thematic sections, non-intuitive navigation, and a text-driven home page.

The two sites are similar in all Engagement Factor-related statistics. Since Yukon Photographers ranks slightly higher in Visits, it is ultimately slightly higher in Engagement Factor.

Staying in Tune [EF 14^th] / Astonishing World of Musical Instrument [EF 35^th]

Although music seems to be a popular topic for users of the VMC Web site, two Virtual Exhibits about musical instruments, Staying in Tune and The Astonishing World of Musical Instruments (AWMI), have very different Engagement Factor rankings. Staying in Tune ranked 14^th and The Astonishing World ranked 35^th.

Possible reasons for discrepancies in EF

Astonishing World makes an attempt to explore how music is an essential element in the understanding of ‘primeval myths’ however there is no introduction to individual people or cultures. AWMI is aesthetically attractive and impressive in design, but the bulk of site content is created in images (GIF). Many pages contain one large image and not much text. This means that the images are saveable, but users cannot cut and paste text from the web site itself. Heavy use of images as content affects the Engagement Factor Ranking because search engine robots cannot index content contained in an image. Designed in a traditional exhibit narrative linear format, AWMI doesn’t allow for multiple entry and exit points (this would affect page ranking in Google). Perhaps most important, unlike Staying in Tune, users are not offered opportunities to listen to the sound of different instruments.

Christmas Traditions in France and in Canada [EF 17^th] / Panoramas: The North American Landscape in Art [EF 20^th] / Perspectives: Women Artists in North America [EF 13^th]

Two very different Virtual Exhibits that rank in the mid-range in Engagement Factor are Christmas Traditions in France and in Canada and Panoramas: The North American Landscape in Art. Closely related to Panoramas is a third Virtual Exhibit, Perspectives: Women Artists in North America. A consortium of international partners developed all three sites. Christmas Traditions in France and in Canada, the first Virtual Exhibit designed by CHIN, was launched in December 1995 as a Canada-France Agreement experiment in museum Internet productions. It is a very popular seasonal Virtual Exhibit, spiking in usage in November and December each year. Panoramas: The North American Landscape in Art and Perspectives: Women Artists in North America were both quite recent collaborative projects among Canada, Mexico, and the United States.

Christmas Traditions in France and in Canada and Panoramas: The North American Landscape in Art are the most popular Virtual Exhibits in terms of number of Visits and Visitors, and two of the most highly ranked in terms Page Views. However, users spend little time on each site. As a result, their Engagement Factor ranks are only mid-range.

Possible reasons for discrepancies in EF

Christmas Traditions in France and in Canada ranked mid-range for Engagement Factor. High seasonal popularity offers little support to the relevance of Engagement Factor for measuring quality in online user experience. It is a Virtual Exhibit with a 1995 design site. The seasonal variation is critical for this site – it has the highest number of Hits and Visitors in December, but has one of the lowest overall rankings for Duration of Visits.

For different reasons, the Engagement Factor does not reflect the high quality design, visually rich images, and multiple forms of communication for users exploring both Panoramas: The North American Landscape in Art and Perspectives: Women Artists in North America. Perhaps in this case, users may require more comfort and expertise interpreting the artworks, which may potentially discourage users who visit the Virtual Exhibits to spend time on them.

The Adventure Train & Safe Trax [EF 25^th] / Hockey: A Nation's Passion & Sign Me Up / Regulation Size Only [34^th]

The Adventure Train is an interactive game about the Canadian railway launched in May 2002. Safe Trax was launched in January 2003 and has been marketed by Sympatico. The Web site is about helping a girl and boy stay out of danger as they venture near a rural crossing, a rail yard, and a railway bridge. Both sites require Macromedia’s Flash player in order to play.

Hockey: A Nation's Passion has more Visits and Visitors than The Adventure Train. Users spend much more time playing The Adventure Train than they spend time on Hockey: A Nation's Passion. Since Sign Me Up / Regulation Size Only, an interactive game on the Hockey Site was launched and featured in late 2002, the Engagement Factor statistics for Hockey: A Nation's Passion have increased.

Generally, the Engagement Factor ranking for both VMC products has remained quite low even though they were featured several times. The Adventure Train was marketed as part of a Sympatico Campaign. Statistics spiked enormously during that period, but returned to the previous levels as soon as the contest was over. One reason for this might be that sites created with Flash are not easily indexable by search engine robots. Thus, once the campaign was over users didn’t have a means to find the site again. ‘Hockey’ as a Canadian sport is featured throughout the VMC Web site’s Landmarks. The statistics on the interactive game Sign Me Up / Regulation Size Only, a component of this Virtual Exhibit, may impact on the statistics for the site.

Museum-based activity and games should introduce youths to concepts or activities that are new to them. The sites should allow young people to engage in a series of actions, which will teach them something meaningful (e.g., about railroad safety). Online interactive games can also creatively educate and encourage discovery of a museum’s collections.