How to Do Usability Testing Cheap and Fast

Bill Moore of RadioTime shared his experience of gathering feedback from users in a sandwich shop. It’s amazing how easy it is to get user feedback with zero overhead, that’s cheap, and leads to actionable results.

I asked him why didn’t he go to the (in)famous coffee shop since their wi-fi usually works well (wi-fi in the sandwich place was flaky). He said the coffee shop is louder. At the sandwich shop, people stay longer and the tables are bigger.

RadioTime.com offers a free complete guide to radio and sells a product that lets you record radio just like TiVO records TV. When doing on-location testing, consider the following:

  1. Pay attention to time and setting: People are less likely to be in a hurry at lunchtime than in the morning before work, for example.
  2. Make testing clear before starting: Be upfront about the time commitment, what you’re doing and why, what the user will do, and any other relevant information. RadioTime lets the user know that there’s nothing to download.
  3. Keep notes: Be ready by having a form and a clipboard. Perhaps, consider recording the user’s feedback with a non-intrusive recorder. Use a clipboard in case you can’t access the table when making notes.
  4. Talk with the store manager: This assures the store that you’re not doing anything suspicious and plus you’re building a relationship.
  5. Wear a logo or display company badge: Helps with trust and credibility.

Here’s what Bill said about doing testing in the sandwich shop.

“The information is especially useful to us because users are not in a controlled environment. They have their own PCs with their own media players, shortcuts, browsers, etc. Not surprisingly, this shows problems that are invisible when testing on our systems.

inside_intuit.jpg“I borrowed the idea from a book on Intuit (Inside Intuit) where they had a program ‘follow me home’ watching customers balance their books. Then the famous Krug book Don’t Make Me Think made the case to test early and often on anyone you find.

“I walk up and buy $30-$50 of gift cards and sometimes find the manager to explain. So the shop is getting a benefit.

dont_make_me_think.jpg“We can test a half dozen people and as many computer configurations cheaply. Ours is easy because it is a general consumer site and it’s easy to get people interested.

The setting and time are important. I wear a shirt with our logo and have a clipboard. Mid-morning or afternoon is best when people are less pressed for time and open to do it. In our case saying we are a local company testing a site helps because they have some empathy and won’t need to download any suspicious software.

“Asking for five minutes upfront is important, sometimes ten minutes depending on what we are doing. Their patience and the incremental value of their testing fades quickly anyway. Like any testing — up front make it clear we are testing the software, not the user. And invite them to think out loud to better understand what is happening.

“I make a form with the main tasks to make it easy to keep notes. It is impossible to both watch and keep good notes by yourself. On the other hand, two people are more intimidating and you can capture the main points later.

“Like any user testing, it can be painful to watch.

“I would like to see more developers go to the shop or watch when we do testing in the office. They are the ones that need it the most to witness first-hand why the details are so critical.”

From a content POV, the feedback is valuable. Users made comments that they didn’t understand what unreliable streaming meant and other similar comments. This helps me identify what needs explaining or rewriting.

2 thoughts on “How to Do Usability Testing Cheap and Fast”

  1. Tell me if I follow correctly: Do you do the test only on users that have their own PC with them? So if you happen to be in a store where nobody has a machine or is willing to take the test, then, tough luck?

    Why worry about having the users test in their own environment (PC)? The focus of the test is on usability. Testing to see if the site works properly across different systems should be part of QA work and can be done separately.

  2. Depending on the store and the time, you are highly likely to find users working on their notebook. So, no computers has never been a problem. The setting and your approach are critical to get to a sure. Mostly, people are empathetic.

    We test with different OS/browsers. But the main advantage of Panera’s (restaurant) testing is getting users on their computer.

    This surfaces plenty of bugs that are unlikely to be discovered with “QA testing.” Watching how people react and recover puts an entirely different perspective on which “bugs” are important. We find far more QA problems in an hour compared to days of brain dead “Does every feature work on XP/Firefox? Yes, Vista/IE, yes…”.

    Also you find a lot of problems that people won’t report because the situation made them feel stupid or somehow responsible.

    Use testing is more realistic than a lab. The user is completely comfortable with their machine. That means testing is on our software in a real-world environment. They don’t stumble, “where is the delete key”, “how do I open the browser again”, “I hate this mouse”, “where did I save that download?”

    You can see if they find our shortcut and tray icons amid a cluttered desktop (as compared to a sparkling clean Vista install).

    You see problems like the Skype IE “phone number dialer” screws up layout (no one would report that).

    All kinds of surprises come up.

    Most important for our product, we see how users react to the software that depends on browser and media players installed. A big part of the test “that we consistently flunk” is helping them get the machine ready.

    We have tested on our machines. My experience is developers especially don’t like the approach above because it’s not focused and there are too many variables. Precisely. The same was even true with the last design/UX team I had. They wanted to focus on “how does this look in Safari”, and tightly controlling testing of a use case. I agree there is a role for that. But what I found is our main problems have been outside of an unrealistic starting point “OK from this screen how would you LISTEN to the program.” “they pressed listen, it worked!”

    Another use testing trick that worked great for us was recruiting from our building. We setup a table in the lobby with coffee and donuts. Then we had a schedule form and offered $25 for 30 minutes. Everyone was in the building, so it was an elevator ride for them. We had a controlled environment and the developers could watch (but most would find an excuse not to). About 50% are no-show.

    We’ve not done any of this in a year and I plan to with the site updates. It takes some time but is priceless.

Comments are closed.