How I Was Studying 3 Buttons for 20 Hours

Let me set the scene. It was during our weekly meeting:

— Andrew, we’ve made a new awesome feature, you should test it
— Yeah, sure
— Hey, maybe you’ll even write an article about it!
— Uh-uh…

There was just one problem…

When I’m asked to test a feature, it usually means conducting a usability survey, which involves finding participants, preparing a set of tasks and questions, and then executing them (the questions, not the participants) in a Skype interview.

So I went to our website to play with the feature myself, imagining different tasks associated with it. It took me 11 seconds to try everything possible. I even tried to do it backward and pretended I was having a conversation with my mother on the phone. 30 seconds. Then my mother actually called me, so I tried it again. 20 seconds.

The feature I’m talking about is these three buttons in the upper right corner. They switch between three view modes.

My usual Skype interview lasts 30-minutes: 80% of it is tasks and 20% is usually small talk and my mediocre jokes. With 30 seconds for tasks, I had to cover the other 29 minutes with my jokes. I wouldn’t do that for 2 reasons:

• No living soul deserves that
• If I could do that, I would be a stand-up comedian

So, all I had for the moment were three buttons that I could test with one question.

My notes at this point looked like:

1. Some task that checks if people see these 3 buttons
2. “Of these 3 view modes, which one do you prefer?”

Looking for More Stuff

I needed something more. So I dug deeper. In one of the modes, I found the 4th button.

1. A task that checks if people see these 3 buttons
2. “Can you tell me what this ‘plus’ button does?”
3. “Of these 3 view modes, which one do you prefer?”

Yet at this point the interview was still very short and I ran out of buttons. I was just sitting and bluntly clicking between all of the three modes, thinking about life, existence, and a nervous breakdown. Then it hit me.

Up until this moment I was thinking about UI, that is an interface. Specifically, a set of buttons and what these buttons were doing mechanically – switching between 3 modes of viewing icons.

Now I was thinking about UX, or experience. A multitude of questions rushed through my head. Do text labels affect people in some way? Does it affect how people search icons? Does it affect their overall experience? What’s for dinner?

All these abstract questions are great, but I cannot simply ask participants “hey, do you find text labels useful?” because there is a big difference between how people answer questions and how they really do things. I was faced with finding concrete tasks to see how people really act.

How? I just got into their skin and start searching different icons. I was searching for car icons, cat and dog icons, popular icons, and uncommon icons. I switched between the modes and searched again and again.

Of course, if you search for different things you get different results. But the real difference was my focus. If I was looking for a cat, I had clear expectations. However, if I looked for “news” or “music,” my search expectations were much more vague.

Even if different search expectations matter, what about different view modes?

This one simple question filled my 30-minute quota.

Moral: always dig deeper

Final Script Draft

1. A task that checks if people see these 3 buttons
2. • SEARCH in [just icons] mode: cat, phone, arrow, news, start, music
   • SEARCH in [icons with label] mode: dog, line, box, idea, stop, work

   Point out irrelevant search results

3. “Can you tell me what the “plus” button does?”
4. “Of these 3 view modes, which one do you prefer?”

I wanted participants to search for a “cat” icon in icons only mode and then point out any irrelevant search results.

Then I would ask another participant to search for the same “cat” icon, but in icons + label text mode, and point out irrelevant search results.

Ok, let’s take a better example. Suppose, you search for a line icon:

With text labels, participants would no longer find the “command line” icon irrelevant. But that’s just one of many effects.

My goal was to prove that different people consider some results more relevant than others, so I’d see the difference in their perception. I would also see if labels affect perception in any way.

Moral: struggle to find ways to measure the experience of your users.

Testing My Test

At this point my script was robust, but I was unsure about it and asked my colleague to act as a participant, as if it was a real interview.
Turns out it was not that robust at all. I had to make some changes.

1. “Can you change the way icons search results are represented?”
2. • SEARCH in [just icons] mode: cat, phone, arrow, news, start, music
   • SEARCH in [icons with text label] mode: dog, line, box, idea, stop, work

   Point out irrelevant search results

3. “Can you tell me, without pressing it, what the ‘plus’ button does?”
4. “Of all the three modes, which one do you prefer?”

Changes:

• The first question led to ambiguity
• With 10 different searches, the whole interview took 60 minutes. I was aiming for 15-30 minutes, so I had to reduce the number to 6
• In the third question, I had to explicitly ask the participant not to touch the button – too much temptation otherwise

Moral: it’s good to do some kind of reconnaissance before you start conducting real interviews

What’s next?

Victory loves preparation. Now I had my script and was ready to test it with real people, so I did just that. 7 participants, 15+ pages of notes and 3 hours of video material that I need to structure somehow. And I’m writing an article about that right now. Next article will be about results, insights and awkward silences. If you are reading this and don’t see a link to the second article, there are two things possible:

1. Current article was released recently
2. I went mad and I’m writing conclusions on my window

I post updates on my twitter.

Tune in!

---

About the author:
Andrew started at Icons8 as a usability specialist, conducting interviews and usability surveys. He desperately wanted to share his findings with our professional community and started writing insightful and funny (sometimes both) stories for our blog.