User Testing Library Search: What’s the Point?

“We must have strong minds, ready to accept the facts as they are, and to make bold, new plans based on those facts.”
—Harry S. Truman

Alchemy, Phlogiston, and Library Science

Eleven years ago, I was carrying around a twelve-pound, thirteen-month old baby that I had just adopted. She was quite sick, and obviously wasn’t eating (she had lost a lot of weight), so we took her to the doctor and were given herbs. They did nothing. We somehow got past the SARS screeners at the Hong Kong airport, made it home, and put her on antibiotics. After a few years of feeding therapy, she finally grew enough to make it on to the height and weight chart, and is now a thriving pre-teen who loves pasta and hates rice, oddly enough.

The unjustified belief in traditional Chinese medicine is a reason why tigers are endangered, likewise why albinos have been murdered in Africa, and why my own health insurance premiums cover the reimbursement costs of snake oil sold under the illogical pretense that water has memory, and that the more you dilute something the stronger it gets.

Human culture is rife with traditions like throwing salt over your shoulder, other pointless sacrifices, and old wives’ tales that have no basis to justify such behavior. Maintaining the ceremonial symbolism of such practices can be meaningful for tradition’s sake, but when you have the equivalent of divining rods being used in a work environment, be it a hospital or a library, that’s bad.

My favorite college professor told me once, if you really want to get to know someone, don’t look at what they like, rather find out what really pisses them off. For me, as an information professional, it’s how people distort facts to support their ideology. When my mom forwards me the latest “Obama’s a Muslim” chain e-mail, it bothers me, not just as the black sheep Democrat of the family, but because it’s so obviously untrue.

Instead of warping reality to match our preconceptions, we need to take a more scientific approach of acting based upon observed phenomena. This cycle of observe-orient-decide-act is the definition of evidence-based practice and the purpose behind user testing.

Companies do this all the time. Think of marketing focus groups, and it’s also why you see those pop-up surveys. There’s a commercial interest in knowing consumer tastes; a company will either be a success or go out of business depending on how well it meets those needs. Libraries don’t often deal with spending behaviors, but to fulfill our mission, we need to learn how to best serve our members.

The affordance of library research tools has changed substantially from early OPACs and federated search engines. As we move towards discovery interfaces and linked data, what were once purely technological impediments are giving way to commercial interests, filter bubbles, and other delivery models that detract value from accessible content. To give library users the best search experience, it remains important to make improvements based on what doesn’t work for them.

Know Your Limits

User testing is not the end all be all for determining what services to offer. As Henry Ford could have proclaimed, “If I’d asked people what they wanted, they would have said a faster horse.” The best argument against usability testing is perhaps a five-minute conversation with the average user. People don’t always know, or properly communicate, what they want. Consider the reference desk transaction where someone asks you for directions to the shelves containing scholarly journals. Although your website should serve those people, you don’t necessarily want patrons like that calling upon their limited expertise to build it.

There is a skewed user base in all of this. It can’t be helped. We have to extrapolate, the best we can, from a self-selected group of participants that is hopefully somewhat representative of the entire population we serve. This is especially problematic when dealing with squeaky wheels. I don’t just mean that single faculty member who has the director’s ear, and if they don’t like something then nobody does, but also the part of your population that frequents a service desk to ask for help. This is the (thankfully in our case dwindling) subset of customers that can’t figure things out on their own. It is safe to assume that the now majority of people who use the library autonomously have less difficulty navigating its interfaces.

In usability studies, subjects are placed in an artificial environment and know they are being watched by people whose products they are asked to critique. Subjects might deliberately or unconsciously adjust their behavior because of this. Those with an axe to grind may similarly be disingenuous about how difficult a website is to use.

We’re also not designing and maintaining a search engine from scratch. There are limited customization options available in vendor-supplied interfaces, which are nowadays rapidly developed and frequently updated. Customer groups even vote on feature enhancements that they would like to see programmed into future versions. This is certainly the preferred way to have adjustments made to licensed products that we pay for. Unless you have in-house developers with spare time, installing add-on scripts that rewrite a system’s core features may create an unsustainable workload.

We don’t, for the most part, customize our ProQuest, EBSCOhost, and other search platforms. A new discovery system is more akin to these hosted services than a local catalog needing custom branding. Just as many libraries fit within a larger organizational mission (that may even mandate a certain website template be used for the greater good), we should start looking beyond our own individual library needs and promote the development of more usable interfaces from content providers, without the need for further tinkering on our end.

All of this isn’t enough to pack it in and not bother, but reasons why we should consider the limitations of any results from user testing. It is best to focus efforts on features related to the basic settings with regard to wording choices, simple styling, and existing configuration options.

Check Your Prescriptivism

The terminology we use is another pitfall. Words like “catalog” and “database” have little meaning to those unfamiliar with the inconsistent naming conventions followed by libraries. Neither is a library search experience, for the typical patron, an intuitive process with perfect results. The need for user education is alive and well.

Given the puzzling jargon and vendor names that we have to work with, it’s obvious that you can’t make anything totally idiot-proof. Our fire alarm goes off about once a week because someone doesn’t notice the big red “Emergency Exit Only” sign on the door. When a significant portion of your user base is using a website in what you call the “wrong” way, however, it’s an indication that the design itself is what’s faulty.

It can be a delicate matter of working with designers to alter their original artwork or wording choices. Librarians in particular can be overzealous about the details of a website’s content and presentation. That is the very reason why you get data from user testing to serve as objective supporting evidence for such requested revisions.

What People (Should) Want

A big challenge in our profession is finding a way to accommodate old-fashioned search behavior while also pointing out that there’s now better ways to do things. Before article indexes existed, the most efficient way to do a subject search was to browse the current periodicals section and flip through journals until you located something on your topic. We are still asked “where are the psychology journals?” by library users who don’t either choose or know to practice newer search techniques, as well as students who have been instructed to do research that way.

Other patrons ask where to execute an advanced search query based on Boolean logic. One of the benefits of a discovery layer, compared to an OPAC, is that in addition to merging articles and books in a single search engine, there’s ways to refine your results. These post-search delimiters are a more efficient way of searching: they’re not always needed, and you can adjust results without having to redo a search completely. Using a new system like this makes for a better search experience, but only once you do it right.

This is an example of how there’s a lot in the library realm of research that is not intuitive, and only makes sense once you learn it, or at least jettison research methods from the card catalog era. A library search interface needs to strike a pragmatic balance in responding to user needs and preferences while also molding efficient behavior, given the available technology.

Logistics

Testing human subjects may fall under the purview of an Institutional Review Board. The determination of what constitutes “generalizable knowledge” is a grey area. We received a waiver that our study did not need IRB approval.

To recruit the best sample population, it’s best to give people something meaningful (e.g., free food and/or gift certificates) to compensate them for their time. At UWM, the procurement process involved a fair amount of red tape, but we were eventually able to offer gift cards to volunteers. We then advertised the study via signs in the library and social media.

We were fortunate to have a campus Usability Testing Laboratory available to conduct the sessions. We worked with them to prepare a list of tasks for users to perform, scripts for introducing the study’s purpose and debriefing subjects, and qualitative follow-up questions.

Moving Forward

Search@UW (Primo-Alma) is the new shared ILS in the UW System Libraries. Our state consortium has historically been a rather loosely-governed group. There is little agreement, or at least acknowledged authority, on system-wide policies. Many campuses continue to work independently, not to deliberately undermine efforts to make centralized decisions, but out of divergent perspectives on how the system should be configured.

One college doesn’t even call it Search@UW, for example, while our flagship campus has sunk costs in maintaining a separate system in parallel. It is hoped that the usability testing results will help promote more consistent and efficient efforts to streamline work on our common system.

Any experiment can be subjected to nitpicking and biased interpretations. Rather than being predisposed to discount data that you might not agree with, it’s important to accept results and take positive action based on them, even if they don’t support your design philosophy. Otherwise you’re just practicing quackery.

Further Reading

• Usability Testing Primo: After the release party, there’s the after party
• Usability Test Results for a Discovery Tool in an Academic Library
• Use and Usability of a Discovery Tool in an Academic Library [paywalled content]
• Web scale discovery: the user experience [paywalled content]
• Do I need to get an IRB Exemption for a Usability Study?
• Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems
• Search@UW (Primo-Alma) Usability Testing Materials
• Why You Only Need to Test with 5 Users
• Modeling a library website redesign process: Developing a user-centered website through usability testing
• The Death of the library website redesign
• Designing the Web with Jeffrey Zeldman
• How a Web Design Goes Straight to Hell