Facet Testing Report from July 2016

Overview
of Findings

The good

• The new facet design greatly increased the visibility and usability of the
filter area. Users in the Control Group took an average of 2 minutes and
30 seconds to find the facet list. Users who saw the updated facet
designs took 25 seconds on average to locate the filter area.

• Both forms of filter logic were easy for users to get the hang of,
although the majority of users expected the new (OR based) filter logic.

• Users of the new design had an easier time 'reading' which filters they
had previously applied to a collection page.

The not so good

• Collection and sub collection hierarchy was confusing for users. Users
did not expect to see (sub)collections interspersed with items.

• Users had trouble returning to the homepage and often looked for a
link back on the far left of the navbar.

• Users wanted more guidance in using Archive.org. Users were unsure of
what types of content to look for and wanted to browse by genre.

Overview
of Findings

High-level themes

• The new filter design greatly decreased the amount of time that it took
for users to locate collection facets and also made it easier for users to
tell which facets were currently applied. All users of the new design said
it was easy to use, fell within familiar or preferred design patterns and
acted as they had expected.

• The new OR based facet logic was slightly more predictable (and
preferred) for first time users. However users were generally flexible and
amenable to which ever facet logic was present in the version of the site
they performed their session in.

• First time users wanted more guidance into the Archive's collections.
Users were quickly overwhelmed by the depth and breadth of the
Archive's items and collections. Users wanted to be able to view
content by entity, event (context) or genre and wanted better signaling
of where 'high quality' or unique items in the Archive might be located.

• The navbar and iconography were hard for many users to comprehend.
Users often had trouble identifying the icons and expected - once
clicked - for the icons to open to a new page or act as a filter.

15 new users recruited from Usertesting.com performed 6
tasks on the homepage and collections pages through an
unmoderated and recorded session.

The goals of the sessions were to:

• Determine which (if either) method to approaching facet relationships is
the most intuitive and easy for users to learn and use.

• Are the filters easily recognizable and findable?

• Can the user successfully utilize filters to locate relevant information?

• Do the facets respond in a way that users expect?

• What are users first impressions ofArchive.org?

• How do users initiate a search for an item of personal interest to them
on Archive.org?

About
the Test

Users were broken up into three groups:

AND Facet Logic

Tested on Tracey's Dev Box
Testing of new facet design
with current facet logic

Mixed OR Facet Logic

Tested on Tracey's Dev Box
Testing of new facet design
with new facet logic

Control Group

Tested on the live site
Testing of current facet design
with current facet logic

Dive-in to
the Sessions

Task

Analysis

Homepage

• The 'About' blurb on the homepage was usually the first element that
stood out to users. Users were attracted to wide variety of media types
present on the Archive as well as the counts per media type.

• The overall look and feel of the homepage and collection pages felt
'drab,' 'dated,' or 'boring' to a number of users (particularly British
users). These comments were particularly focused on the background
and collection tile colors on the homepage.

• Although users liked the images on the collection tiles, some users
expressed that the tiles looked 'messy' or were hard to navigate. Users
wanted the collections on the homepage to be organized either
alphabetically or by subject matter and the infinite list of collections was
overwhelming for some users.

• The Wayback Machine search box in the navbar attracted a lot of
attention from users but was confusing for most users. On first
inspection some users thought it was a banner ad based on its
placement and the design of the logo.

Task

Analysis

Continued

Collections Page

• Collection hierarchy was confusing for users. The relationship between
the collections on the homepage, subcollections, media type based
collections and individual items was unclear for most users.
Subcollections were generally read by users as links to a new page
which would have more information about a single entity, event or
genre along with relevant items.

• The relationship between items in a collection was unclear on
Community collections and the Etree collection (most users did not
recognize any of the bands on the Etree collection and were therefore
unsure what the collection was built around). Users wanted more
information about a collection on the collection tile and at the top of
the collection. Very few users navigated to the About tab.

• Users had a hard time exiting the collection page. Those who wanted to
see more items of the same mediatype appeared to be unsure of where
to navigate to. Many users returned to the homepage to begin a new
search.

Facets

Task

Analysis

Continued

• Case-sensitive facets (Rock vs rock) were frustrating to users who
wanted to see all of the items associated with a specific topic or
subject. Even with the OR facet logic in place, users still had trouble
ensuring that all of the relevant facets (upper, lower, mixed case) were
selected. This lead to only a subset of all results being exposed.

• Users quickly found the 'More' link, but were overwhelmed by the total
number of facets. The original design for this page is a paginated
model, something a users suggested during testing. Additionally, users
expected this list to be in alphabetical order.

• The new facet design greatly increased the visibility and usability of the
filter area. Users in the Control Group took an average of 2 minutes and
30 seconds to find the facet list. Users who saw the updated facet
designs took 25 seconds on average to locate the filter area.

• The new OR based facet logic was slightly more predictable (and
preferred) for first time users. However users were generally flexible and
amenable to which ever facet logic was present in the version of the site
they performed their session in.

Task

Analysis

Continued

Free Explore Time

• First time users wanted more guidance into the Archive's collections.
Users were quickly overwhelmed by the depth and breadth of the
Archive's items and collections. Users wanted to be able to view
content by entity, event (context) or genre and wanted better signaling
of where 'high quality' or unique items in the Archive might be located.

• Many users began their free explore time by looking for a way to
browse content within a specific media type. Users were quick to find
the icons on the navigation bar, but were surprised that the icons did
not lead directly to 'media specific' pages (eg. images.google.com or
books.google.com) . Additionally some of the icons were confusing to
users especially Audio and Software.

• Community and media type collections were often overwhelming for
users, who wanted to be able to browse the collection as a way to learn
what types of items might be included in the collection. Commonly
they looked for signals based on genre, similar to a user browsing on
Netflix.

Key Insights
and Findings
Continued

Media Type Filtered Search

Although users generally had a positive reaction to the wide variety of
media types featured on the Internet Archive, users - especially during the
free explore task - looked for ways to limit their browsing and searching by
media type. These were users who had successfully utilized the collection
facets and had noted the ability to filter by media type and yet they still
looked for ways to narrow before beginning their search.

Recommendation

Q v Search

Search the Entire
Internet Archive

Art / Images
Collections

L/al9

eBooks

Events

Geography

Music

People

Saci a MoHia

JUUol ivitru la

Television

Websites

1 1

Advanced Search

Enabling users to quickly
narrow down their search
from the search bar could
help users feel confident
and in control of their
searches and reach relevant
information faster. This filter
area could also be used to
surface advanced search
throughout the website.

Suggested ‘Search Filter’ Design

Key Insights
and Findings
Continued

Create a Home for Each Media Type

Most users began looking for an item of interest to them by looking for a
way to browse content within a specific media type. Users looked for a
page that was specific to a single media type as a way to specify before
exploring by genre, event or contributing institution.

Recommendation

Web Collection

Search and browse 15 years of .atl sites

Visual Showcasing of Media Type Pages on the Homepage

Create a 'home' for each media type that is visually more similar to a
homepage than a collection, including visually highlighted collections or
genres as well as featured content. Link to these pages through a
showcase on the homepage and through a redesign of the navigation bar.

Key Insights
and Findings

Highlighting of Unique and High Quality Content

The amount and breadth of the items available to users on the Archive was
exciting but often overwhelming for the majority of users who were unsure
of what specifically the was available through the Internet Archive.

Recommendation

Volta

by BfOf*

~ Atlantis CD Coll
^ _ hlaliOftdJ Library

Sugar Cubes in Japan
Photos

by Will Strands

H University of Atlantis

Dancer In The Darfc
Behind the scenes

by Lars Von Trier

Univ

Johr

Paper Magazine Photo
Shoots

FEATURED ITEM

Stonemilker Virtual Reality Video

Pea horseradish azuki bean lettuce avocado asparagus okra.
Kohlrabi radish okra azuki bean com fava bean mustard
tigernut jkama green bean celtuce collard greens avocado

Suggested ‘Featured Item’ Design

Create a new Ul element that would enable
Internet Archive staff, partners and
collection creators to highlight and annotate
featured items in a collection.

These items could appear mixed in among
other items in the collection and serve as a
way to guide users to special items. The
space for annotation would allow for
contributors to share what makes this item
noteworthy. These featured items could also
be displayed on the homepage, as a way to
browse across collections.

Key Insights
and Findings
Continued

Collection and Sub Collection Hierarchy

The intermixing of collections and items on collection pages was
disorienting for some users

Recommendation

Media Type

20 COLLECTIONS

texts 1524

memos 632

audo 323

collections 20

Dpics & Subjects

Cnctai
Yoga
Cats
Dance
Astrology
lore ►

1843

60S

564

548

473

flection | &»

HoOoos and Hifinks 302

A Brief Exp oration of 244

Cat Naps

The Cactus Cryptograph 235

Trouble with Tabbies 1 86

Contemplate the Catn p 1 43

Hodooa and Hijinka

- . u\n .

3.932 ITEMS

A Brief Exploration
of Cat Naps

The Cactus
Crytcgraphy

Trouble with Tabbies

Contemplate the
Catnip

Title > Date > Creator

IkhfiOT rfT Yngift: l

by Of Mmu W#»t

■ 5 GRID = LIST

\m

' £% -

I Hi

Suggested changes to the collection page

In this design sub collections are separated from item results on the
collection page. Additionally collection tile height is standardized as some
users thought that larger collection tiles were a sign of a larger collection.

Key Insights
and Findings
Continued

Navigation Bar + Iconography

Some users had trouble returning to the homepage once they navigated
away and often looked for a link back on the far left of the navbar. More
generally, the Top Hat' (clicking on a item opens above the navigation bar)
was disorienting for some users, who expected the navigation bar to have a
dropdown. Additionally, some users expected clicking on a media type icon
to open to a new page or act as a filter over the current page.

Legibility of icons may also be a problem as users relied primarily on 'helper
text' to decode icons.

Recommendation

Our navigation bar is burdened by the quantity of content that needs to be
accessible through it. Many other websites face similar challenges and
looking at how they approach this design problem could help to influence
further designs on the Internet Archive (please see the next slide for some
examples). Higher visibility (expansion) of the search bar, a 'sticky'
navigation bar, reorganization of submenus and increased text (over
iconography) should be explored.

Examples of High Content Navigation Bars

Key Insights
and Findings
Continued

Bloomberg the Company A Its Products

Bloomberg Anywhere Remote Login | Bloomberg Terminal Demo Request

Bloomberg

Businessweek

Tech

Pursuits Politics

Opinion

Sign In
Subscribe

Stocks Currenc

Regional Overview
Stocks

Stock Futures
Gainers ♦ Losers
Company Profiles
Earnings Calendar

Commodities Rates 4- Bonds Economics Magazine Benchmark Watchllst Economic Calendar

Dow Jones Indus. Avg SAP 500

FTSE100

-0.60 % -0.46 o/o -0.43

Nikkei 225

New York
Public

Library

LOG IN -

LOCATIONS

GET A LIBRARY CARD GET EMAIL UPDATES

DONATE

SHOP

Browse

Learn Attend Research Give Get Help

Browse

LOG IN

RECOMMENDATIONS

Books/Mustc/DVDs

Recommendations

4ft

MyNYPL

Staff Picks

Blog

Public Projects
Shop

Log into your account to check holds, item
due dates, and more.

There is a huge world of books out there.
Our expert staff members pick out their
favorites to help you find your next one.

These two navigation bars from Bloomberg and NYPL, are content heavy
but create clear pathways for users to access relevant information through
color contrast, space between elements and clear navigation indicators.
The Bloomberg navigation bar also includes a top bar connecting all
Bloomberg web properties together through a single navigation system

Additional Testing with Current Users

Testing the two facet logics with current users of the Internet Archive
would help to better understand the types of tasks that expert users of the
site rely on collection and search facets for. The new users tested in this
round of testing for the most part seemed amenable to either facet model,
as both are models they come across while using the internet. It should be
stated that the OR model seemed to be more expected by new users,
however input from current users could help to determine which model is
a better fit for Archive.org.

A/B Testing of Both Facet Logics

An alternative to more formal testing with current users, or incase current
users are still split between AND and OR facet logic, live A/B testing of
both facet logics would be helpful. Success metrics would need to be
defined in order to determine which logic was the most successful.

Next Steps
for Collections

Questions, comments, or concerns?

Please don't hesitate to reach out to me on Slack,
in person or by email at carolyn@archive.org