Dennison asked excellent questions in response to How to Do Usability Testing Cheap and Fast:
Tell me if I follow correctly: Do you do the test only on users that have their own PC with them? So if you happen to be in a store where nobody has a machine or is willing to take the test, then, tough luck?
Why worry about having the users test in their own environment (PC)? The focus of the test is on usability. Testing to see if the site works properly across different systems should be part of QA work and can be done separately.
Bill Moore from RadioTime responded:
Depending on the store and the time, you are highly likely to find users working on their notebook. So, no computers has never been a problem. The setting and your approach are critical to get to a sure. Mostly, people are empathetic.
We test with different OS/browsers. But the main advantage of Panera’s (restaurant) testing is getting users on their computer.
This surfaces plenty of bugs that are unlikely to be discovered with “QA testing.” Watching how people react and recover puts an entirely different perspective on which “bugs” are important. We find far more QA problems in an hour compared to days of brain dead “Does every feature work on XP/Firefox? Yes, Vista/IE, yes…”.
Also you find a lot of problems that people won’t report because the situation made them feel stupid or somehow responsible.
Use testing is more realistic than a lab. The user is completely comfortable with their machine. That means testing is on our software in a real-world environment. They don’t stumble, “where is the delete key”, “how do I open the browser again”, “I hate this mouse”, “where did I save that download?”
You can see if they find our shortcut and tray icons amid a cluttered desktop (as compared to a sparkling clean Vista install).
You see problems like the Skype IE “phone number dialer” screws up layout (no one would report that).
All kinds of surprises come up.
Most important for our product, we see how users react to the software that depends on browser and media players installed. A big part of the test “that we consistently flunk” is helping them get the machine ready.
We have tested on our machines. My experience is developers especially don’t like the approach above because it’s not focused and there are too many variables. Precisely. The same was even true with the last design/UX team I had. They wanted to focus on “how does this look in Safari”, and tightly controlling testing of a use case. I agree there is a role for that.
But what I found is our main problems have been outside of an unrealistic starting point “OK from this screen how would you LISTEN to the program.” “they pressed listen, it worked!”
Another use testing trick that worked great for us was recruiting from our building. We setup a table in the lobby with coffee and donuts. Then we had a schedule form and offered $25 for 30 minutes. Everyone was in the building, so it was an elevator ride for them. We had a controlled environment and the developers could watch (but most would find an excuse not to). About 50% are no-show.
We’ve not done any of this in a year and I plan to with the site updates. It takes some time but is priceless.
Meryl’s comments
We have a few computers in my house and each one works / reacts differently to RadioTime especially where audio players come in. They have variations of browsers and versions installed. One has IE 6 and Firefox 1.x. Another has IE 7 and Firefox 2.0.
What Radiotime has that many typical sites don’t have is Internet audio that relies on the user’s installed media player.
Add browsers to the equation. Windows Media rarely struggles in Internet Explorer, but might make trouble within Firefox. Sometimes Firefox doesn’t handle an already installed plug-in correctly because the user hasn’t installed it from within Firefox. Crazy, but true.
Bill,
I appreciate how you shared your experience and compared traditional QA with this new and dynamic approach. I now see the value of your sandwich shop method as it contributes a sense of relevance and ownership that is absent in a controlled environment.
Could not help but smile on the Skype phone number dialer. I use Skype and the first thing I do is disable the Firefox and IE plugin. I guess that being in the IT industry, it is inevitable for us to unconsciously “optimize” our systems by removing unnecessary clutter or by not installing them at all — something that everyone else tends to do.
I understand how approach is key to success in getting users to participate. I run a store that sells tea (cold and brewed) and one thing I learned is that atmosphere (ambiance, crew friendliness, store cleanliness) contributes a lot to the customer’s level comfort level, which is relative to their mood levels.
It is interesting that you used the same coffee and donuts setting that I’ve seen being done by television and radio stations to gather viewer response. Based on my experience the gem in this approach to information gathering is not the answers to the survey itself but on what happens in between where the user shares personal views and comments — information that would otherwise not have been anticipated by the in-house team.
Meryl,
You are right on the money on the importance of the installed media player plug-in, and I agree with you. On my work machine I use Firefox 3 Beta 2 as my main browser with Foobar2000 as my default media player.
When a site breaks, most of the time at least I understand why (e.g. Firefox is causing it or I do not have the required plugin) but that is because I am in the IT industry. I have 3 machines at home and one of them is running on Linux. How many others have unorthodox setups? I’m sure there are plenty.