Evaluation of the Test Sites

Methodology

Twelve volunteer evaluators reviewed the test sites. The evaluators' primary applications for online access to historical newspapers are historical research (including local history), public library reference, and genealogy.

The evaluators were provided with survey forms that listed major features at each site and click-by-click instructions for navigating the site to find the features. They then were asked to rate the features:

Additionally, the survey forms encouraged free text comments on all features and ratings, and asked for features or performance issues overlooked by the survey form.

After reviewing all sites, a final survey was administered to gather summary information about preferences among features, performance issues, and recommendations for further action.

Findings

The following features were evaluated by at least half of the reviewers as "essential."

Essential features

• Search type (any words or exact phrase on full text or titles only)
• Search dates (start and end dates)
• Search results--essential (publication name, date, and link to article; see also "search results-important" below)
• Locating hits in context (easy and legible identification of search terms in article context)
• Navigating within page (ability to read article in context on page and to move around page scanning text)
• Navigating between pages (ability to move forward and backward among pages of the issue)
• Browsing among dates (ability to identify online holdings for each title-- year/month,/day-and to browse forward and backward among issues
• Text legibility (presentation of text images to facilitate reading, including unbroken images, good contrast and edge definition, and uncluttered background)
• Ease of use of site (uncluttered graphics, easy to understand features with intuitive icon labels, minimal clicking to desired results, maximum screen real estate devoted to text images rather than to feature icons and instructions)
  

Important features

The following features were evaluated by at least half of the reviewers as "important."

• Search article type (limiting searches to news, advertisements, classified ads, births, marriages, deaths)
  
• # of documents to retrieve (limit the number of hits in response to a search)
• Search results--important (elements to be included in the summary listing):
  • number of documents retrieved
  • article type
  • article title/beginning text
  • relevancy ranking by # hits in article
    

Not important features

The following features were evaluated by at least half of the reviewers as "not important."

• Language (to limit searches)
• File size (for each document retrieved from a search)
• Thumbnail images (of articles retrieved)

Preferences among implementations

Preferences varied among evaluators for implementation of "essential" and "important" features. In the following table, preferred implementations are indicated by asterisks; an asterisk in a cell means that at least half of the evaluators rated the implementation of a particular feature at a particular test site as meeting expectations "very well."

[see attached Excel file "preferences"]

"Quick picks" among current alternatives

If there will be NO further development of the test site software, the reviewers recommended several sites for further consideration:
TechBooks/Newsbank/iArchives (6 recommendations)
Cold North Wind (3 recommendations)
OCLC/Olive (3 recommendations)
BYTEMANAGERS/iArchives (2 recommendations)

If a FEW CHANGES could be made to the test site software, the reviewers recommended several sites for further consideration:
TechBooks/Newsbank/iArchives (6 recommendations)
BYTEMANAGERS/iArchives (5 recommendations)
Cold North Wind (4 recommendations)
OCLC/CONTENTdm/iArchives (4 recommendations)
OCLC/Olive (3 recommendations)