Jakob Nielsen admits in his article, "Why you only need to test with 5 users" that 5 test subjects can find 80% of the usability issues; 15 users will find all issues. I'll be honest, based on my experience, that extra 20% of usability issues are usually so minor that they don't have an impact on the main flow. So if 5 test subjects find 80% of the usability issues, why do we keep having tests with 10 people or more? Sure, we want to find as many issues as we can with an online application or site, but why be so complete right out the gate?

A huge risk that should be considered during testing is what if the system as designed just generally doesn't work – people don't get it, they don't like it, find it problematic, etc. and you have to redesign it entirely. At that point, knowing about misspellings and incorrect page space usage doesn't matter. So let's say you have a very complete test group of 15 as Jakob notes in his article, and you find out that the general design doesn't work. You go back to redesign your application. In theory – you should test it again (most times, this does not happen. But ideally, it's a must happen because it is a new system and it's not tested – it's a result of a failed system in the first place). But there is the next risk for the UX – will the business fund testing with another 15 users?

Let's say instead, you approach this as an iterative test as Jakob suggests. You do some work, create a prototype or some type of sketch to get feedback and show it to 5 people. Let's say it's a bust and you go back to the drawing board. You do the same thing again and test with 5 more people. Let's say this time it works, but needs some work. You refine it and then test it again with 5 more people. Let's say there are minor problems to fix. You could do at least 1 more test (if not 3 more) based on the original budget for testing to capture all usability problems. This is far more efficient and effective to product a UX that users will find helpful and they will use.

Every time you make a change in the test, in a way, it's a brand new test because there are different risks and factors in the design that need to be reviewed. I think this is a better method for testing mainly because you can experiment and get constant feedback from your users. Also, you start treating users as more of a stakeholder in your project/product. Right now, users are treated as a visitor – someone who comes in and uses what you made here and there. They usually are involved only 2-3 times in a project. It's up to the UX professional to know their needs or raise questions to get the business owner to know the needs. In a way, that's just a ridiculous notion – we aren't mind readers and we need the direct input of users – just a better communication chain.

The best way to get feedback from users is to include them in reviews, in the same way that the developers have reviews with the business owners. This is the 5 person usability test process. I wouldn't suggest having a board or panel to do the reviews regularly – you could do that to help get new features to add, but usability works best with new eyes. Automated testing with thousands of users is nothing more than A/B testing – and in a way, that is measuring preference (I'll be writing more about this later – but this is also a type of usability that can be tested). Doing an online test for the initial testing sessions is more than sufficient. At those stages of testing the goal is mainly to understand if the flow works or if people can find what they need to complete a task. If you feel the need to have in-person sessions, I would suggest to leave it at the end when you want to see how the user responds to the look/feel of an application. But in general, if you can see the face of a person and their facial expressions, have that person test on a computer he plans to use at the time of day that person will use it, then you should be ok.

Basically to get requirements from users you need to do the hokey pokey – put a hand in (test), pull it out (refine) put a foot in (test), shake it all about (larger test or maybe bigger test). This is probably one of the easiest ways to get the requirements out of users and have a successful product.