Jakob Nielsen admits in his article, "Why you only need to test with 5 users" that 5 test subjects can find 80% of the usability issues; 15 users will find all issues. I'll be honest, based on my experience, that extra 20% of usability issues are usually so minor that they don't have an impact on the main flow. So if 5 test subjects find 80% of the usability issues, why do we keep having tests with 10 people or more? Sure, we want to find as many issues as we can with an online application or site, but why be so complete right out the gate?

A huge risk that should be considered during testing is what if the system as designed just generally doesn't work – people don't get it, they don't like it, find it problematic, etc. and you have to redesign it entirely. At that point, knowing about misspellings and incorrect page space usage doesn't matter. So let's say you have a very complete test group of 15 as Jakob notes in his article, and you find out that the general design doesn't work. You go back to redesign your application. In theory – you should test it again (most times, this does not happen. But ideally, it's a must happen because it is a new system and it's not tested – it's a result of a failed system in the first place). But there is the next risk for the UX – will the business fund testing with another 15 users?

Let's say instead, you approach this as an iterative test as Jakob suggests. You do some work, create a prototype or some type of sketch to get feedback and show it to 5 people. Let's say it's a bust and you go back to the drawing board. You do the same thing again and test with 5 more people. Let's say this time it works, but needs some work. You refine it and then test it again with 5 more people. Let's say there are minor problems to fix. You could do at least 1 more test (if not 3 more) based on the original budget for testing to capture all usability problems. This is far more efficient and effective to product a UX that users will find helpful and they will use.

Every time you make a change in the test, in a way, it's a brand new test because there are different risks and factors in the design that need to be reviewed. I think this is a better method for testing mainly because you can experiment and get constant feedback from your users. Also, you start treating users as more of a stakeholder in your project/product. Right now, users are treated as a visitor – someone who comes in and uses what you made here and there. They usually are involved only 2-3 times in a project. It's up to the UX professional to know their needs or raise questions to get the business owner to know the needs. In a way, that's just a ridiculous notion – we aren't mind readers and we need the direct input of users – just a better communication chain.

The best way to get feedback from users is to include them in reviews, in the same way that the developers have reviews with the business owners. This is the 5 person usability test process. I wouldn't suggest having a board or panel to do the reviews regularly – you could do that to help get new features to add, but usability works best with new eyes. Automated testing with thousands of users is nothing more than A/B testing – and in a way, that is measuring preference (I'll be writing more about this later – but this is also a type of usability that can be tested). Doing an online test for the initial testing sessions is more than sufficient. At those stages of testing the goal is mainly to understand if the flow works or if people can find what they need to complete a task. If you feel the need to have in-person sessions, I would suggest to leave it at the end when you want to see how the user responds to the look/feel of an application. But in general, if you can see the face of a person and their facial expressions, have that person test on a computer he plans to use at the time of day that person will use it, then you should be ok.

Basically to get requirements from users you need to do the hokey pokey – put a hand in (test), pull it out (refine) put a foot in (test), shake it all about (larger test or maybe bigger test). This is probably one of the easiest ways to get the requirements out of users and have a successful product.

 

How do you best learn about animal behavior – move the animal to a lab or go to the environment and observe the animal?

I've been reading a lot about usability testing lately and I started wondering, when we test applications with users – do the users need to be onsite or could the test subjects be somewhere else, like their home or business environments? Based on my experiences with testing, I'm wondering if the model we currently use for testing gets accurate results of if, well, we need to do something differently to really understand how users think.

I took a class a while back about design research. Design research, which is similar to usability research in many ways, has been moving toward having in person interviews with subjects at their native environments (home, work – anywhere that they want to learn more about the user/subject). So let's say for a consumer product to use at home, a researcher would come into your home, watch what you do, and come up with ideas for a product based on your behaviors. This helps researchers understand their audience far better than any interview over the phone or in a lab could do – you get to see how people act directly in the environment they will use the device or equipment.

I started wondering if it made sense to do this for usability testing. I have a few reasons for this:

• When we bring users into a lab, are we really getting a true response as when the same user is at home? Are we really seeing the true response to the user experience? For example, people have a tendency to multi-task and being in a lab, you just don't see that
• How honest are people really when they are being interviewed in person (even if they are told that they are not offending people with feedback)?
• Words vs actions – which means more in usability and user experience. I'd rather see the user's actions than hear them rave about an interface

 

User in a lab vs home – which provides more accurate feedback?

If you are testing a consumer product, you may want to test it while the person is on his home computer. You would want to see if the person does indeed multi-task while using your application, or if the person gets distracted and stops using it for any length of time. What if the user gets interrupted by his kids frequently – it would be good to know that to be sure to include longer session times. What if if the user makes dinner while computing – that would encourage extended session times as well or encourage the development of a recipie program. What about the living room? The bedroom? The bathroom? (ok, maybe I don't care to know the user uses his computer in the bathroom). This is how we have movies, music and books on the iPad and other devices for entertainment.

The same is true for business users for applications – what if the business user gets interrupted frequently? What if he already has 3 other systems to use for a similar purpose? You may get this information from a lab session or phone call with questions, but the best way to learn this is through direct observation of the person's behaviors in their own work environment.

If a user comes into a lab with the 2-way mirror or recording devices, will you get the same response as if you saw how the user works in his natural environment? From experience I have picked up more by being at someone's workplace to pick up on nuances that I never would have seen or experienced in a lab. For example, to watch a user switch application screens very quickly on a 15 in monitor allowed me and a colleague to make a recommendation to management to have data entry clerks get a larger screen to display up to 3 application screens at once. In another case, I heard a lot about how travel agents had a difficult time logging into this travel application. Sure, it sounded bad, but I didn't feel their pain until I sat next to one of them and almost threw the computer out the window myself out of frustration on not being able to log in.

The value of seeing users in their native environments is immeasurable. I see it as a requirement for accurate user testing and research. Otherwise, the test is too clinical and isn't really giving accurate readings except to know how someone uses an application when he is in a room, alone, with no other applications on the screen.

Honesty of test subjects

I think this is always tricky. You never know if someone is telling you the truth in testing about their experiences. You can only get a good idea based on the user's actions and what you observe from the system. With most people, just in general, if their actions and words don't agree – watch their actions. Actions definitely speak louder than words.

Sometimes users will over-exaggerate or over-emphase an issue because they are looking for something to say. They believe that you are looking to get feedback and feel they need to give you something for the money they are getting. Sometimes users don't know what to say – they get how the application works, find what they need, and are not sure what else you are looking to learn from them. There is sometimes a feeling that the subjects know that they are subjects, like mice or rats or monkeys. I've been behind the wall or as a facilitator and heard from subjects "Is this what you are looking for?" "Am I being helpful?" "Did you hear that back there?"

With comments like that, are you getting accurate comments and feedback? Or are you learning about a user in the same way a researcher learns about a rat (getting clinical details that help with research, but they are controlled to find basic items being researched – nothing about the environment).

Words and Actions

As I wrote earlier – you can always trust actions over words. So, can you really trust what someone says about an application if, at the end of the day, he doesn't use the tool anyway? I've witnessed users not reading instructions, skipping through an application, and then at the end saying "You really need to add instructions," as if the user would read it anyway. I've seen users comment how they would buy a product from the company I was doing testing for and make the results work to support that – but their actions showed how well or not well they understood the application.

This is where leveraging Web stats and other tools help researchers understand what users actually do rather than what they SAY they do. It also helps for observation and understanding why users may drop off at a key point. With these theories, you can then make observations that could be tested and researched in the user's main environment.

 

These factors are why I find great success with remote testing in the user's environment. If you have a camera, can see the user's facial expressions, see the user's response to the test questions, observe what they click on screen and see the recording, get an idea how users multi-task, and see what is going on in their home – you learn a lot more about users in a way that is not direct and in their faces. Hopefully, the user will forget that you are there and observing them. I think moving to a model like this – more remote observation in the users environment – will give better results for better product development and innovation.

So I guess this means I would prefer watching gorillas in the jungle to better understand what they do and how they think. I'd rather see how they play and make observations than construct a test that may direct results.