I’ve conducted around 50 usability studies at Icons8. While I was learning my craft I read many articles on the topic. What I found is that some practices are blindly borrowed from laboratory research while others are heavily misinterpreted. And some are pure dehumanizing.

I’d like to address some of the modern usability myths in this article in comparison to my own experiences.

Myth #1 Do Not Listen to Users

Whoever said that was… Jakob Nielsen, one of the pioneers of usability research. Thus the first rule of usability was born. Of course, getting feedback based on real interaction with a product is paramount, but that doesn’t mean usability specialists shouldn’t listen to people at all. The question is, when? Sure enough, this question is answered in the very same article where this first rule was minted.

“When should you collect preference data from users? Only after they have used a design and have a real feeling for how well it supports them.”

What can you get by listening to people that is hidden while they are doing tasks? When I was studying our request icon service:

• people perfectly coped with all the tasks
• people liked the new, modern design

Everything was fine, but one thing. After finishing the tasks, they asked “what’s this whole interface about?” Now, after I heard more or less similar questions from several attendees I got goosebumps. Thus, the infamous 47% story was born.

Summary: listen to people after they are done with tasks. There are at least 47 reasons to do that.

Myth #2 A Hundred Opinions are Better Than Five

Whoever said that 100 people are better than 5 has never been stuck on the subway. To rephrase, the myth is that “automated large-scale tests are better than live interviews.” Automated tests are a good tool, but they are not superior.

First, it takes real skill and practice to interpret big chunks of data in the right way.

Suppose I have an egg farm. If I got a report that 10% of our eggs are cracked, what should I do?

a) raise a number of chickens to cover the shortage
b) focus on the safety of existing chickens to reduce losses
c) fire my cousin

Big data makes us very confident but doesn’t save us from ambiguous interpretations. In personal interviews just one participant is enough to shatter my inner world, leaving me in constant doubt about everything I thought I knew, and, eventually, make me more objective.

I’m not encouraging you to interview every chicken to solve the farm problem (3-4 would be enough). I was making a point – automated test results can be interpreted in a more ambiguous way than results from direct interviews.

Second, one-on-one interviews give so much unique information if you really listen to people.

Summary: Both automated tests and live interviews serve a different purpose and are not superior to each other, but complementary.

Myth #3 Do Not Change the Script

If you tear your notes up 2 minutes before a presentation it’s called courage. If you edit the script while shooting a movie it is called a unique vision. But if you change the script during the usability interview it’s called “WTF is he doing?”

Why are UX people so afraid of changing the script along the way? Because the results will not be objective, they say. You need 5 people to do exactly the same thing to draw some reasonable conclusions about the interface being tested.

While I agree on getting the most objective feedback possible, I disagree that words in your script should be carved in stone. In one of my studies, I watched how people devotedly ignored one button they all saw from the very start. If I hadn’t come up with an additional question, “Did you notice this button before the task?”, I wouldn’t have ever understood the real reason behind such strange behavior.

Another argument for having a rigid script is that it adds comfort to everyone – a participant feels like an interviewer knows what they are doing, and the interviewer feels like they know what they are doing. What can I say? Genuine information is more important than comfort for UX professionals.

Summary: Having a script is very handy, but don’t be afraid of adding to it on the way, just make sure it doesn’t disrupt the overall flow of the research.

Myth #4 Don’t Talk to Participants

A typical laboratory setup is a chair, a table, and a participant sitting at the computer, doing tasks. Cameras and sensors tracking everything – eye movements, facial expressions, body language. And no one around. The goal is to exclude the experimenter’s influence on the participant, leaving them one on one with a product being tested.

For some reason, several authors suggest creating this very lab atmosphere every time, even in more informal Skype interviews. “Don’t talk with users, listen.” While I agree that researchers should listen more than talk, they don’t have to treat their participants like Wolverine in his early lab days.

Even if you are silent, in another room or whatnot, people know they are being watched. You don’t even have to do anything, the experimenter role is your default one. And when you play the role of an experimenter, these are some possible effects:

• participants see you as a figure of authority and they try to please you with their answers and actions, thus bending the truth
• they fear admitting mistakes in UI because they are afraid to look silly in your eyes
• they adopt a behavior model that is different from when they really use your product (at the office, at home)

So take a more friendly role. I always liked making smalltalk with attendees. A few jokes (better funny), a few side questions, a relaxed tone, and people are more willing to share everything that’s on their mind. So you get valuable information about the product, things that might otherwise be outside of a participant’s comfort zone.

Summary: Avoid being too talkative, and strive to create an authentic atmosphere.

Myth #5 Common Sense Isn’t Enough

Usability is a topic everybody has an opinion about. Everyone from your boss to your unemployed cousin. That doesn’t mean there are no real usability professionals. There are quite a few things usability professionals may be familiar with:

• social psychology
• behavioral psychology
• design basics
• communication skills
• data management
• a lot of practical experience

Such a specialist is quite costly, thus many companies take another approach. They simply assign this role to someone. And if that someone has common sense, it pays off.

If you can’t afford an experienced usability specialist but desperately want to test your product, why wouldn’t you take the same approach and become that someone?

A foundational  design principle is that 80% of people will use 20% of the product, it’s “core.” Define the core and come up with a few tasks affecting it. Then watch how people do them. Never give hints, just wait patiently until they ask for your help. Thus you’ll see the real problems and benefit from the research much more than by simply adhering to someone’s subjective opinion.

Summary: Experienced professionals will benefit everyone, but having someone with common sense is a good start.

Conclusion

Usability myths I addressed in this article are more or less directed towards eliminating the “human factor,” to make all usability studies and services as impersonal as possible. From one point of view, that’s correct – in real life, there is no experiment between a product and a person using it. Imagine every time you used a new shower someone came up with a notepad and asked how the water was.

But trying to “hide” the experimenter from the experience results in the opposite—people behave in a way they would never behave in real life. The best way to eliminate the experimenter is to observe the “user” in their natural environment, a full-scale surveillance program, and I doubt the usability industry would be the first to get their hands on it.

So, what’s left? Creating a friendly, trusting atmosphere during research. Not eliminating the human element, but adding to it. Don’t pretend as if you’re not there—at least if we’re talking about a non-laboratory environment, like Skype sessions. But who knows how the same approach would work in a laboratory as well?

By the way, here’s in one of my very first usability studies: Drag and Drop vs. Click – Are They Rivals?

I’m not the only one who uses uncommon approaches in his work—meet Ivan, founder of Icons8, and his Blind Method for hiring new people.

About the author:
Andrew started at Icons8 as a usability specialist, conducting interviews and usability surveys. He desperately wanted to share his findings with our professional community and started writing insightful and funny (sometimes both) stories for our blog.