This post is part of the Pride & Paradev series.
Should you involve real users in testing?
Yes, involve real users in testing
Unless you involve real users you risk releasing something into production that is not user friendly. User testing doesn’t need to be expensive; you can conduct it in house with a focus on simplicity by reading Steve Krugg’s excellent how-to guide Rocket Surgery Made Easy.
The caveat is that you need to conduct user testing as earlier as possible: there is no point doing user testing if it doesn’t result in useful change, and the earlier you do it the more likely you are able to change.
You don’t even need to have high-fidelity screens to conduct user testing: low fidelity screens are often enough to get critical usability and concept feedback from real users.
No, don’t involve real users in testing
The problem with involving real users in testing is the risk that your organization will use this as an opportunity to listen to what users want.
Listening to what users want is dangerous. This is because users think of things in incremental rather than revolutionary terms. Users don’t know how to ask for something they’ve never conceived of. Listening to feedback from users is going to yield incremental improvements (‘make that button green’), but this does in no way correlate to releasing something that users actually want.
In my pre-MP3 university days, I had a very large collection of audio CDs. I worked hard in a casual job and saved up and bought a Pioneer 25 CD stacker: it was amazing, I could listen to 25 CDs on shuffle! If I was asked what could have made listening to music better I would have said a 100 CD stacker: I could store four times as many CDs! A few years later Apple released a new thing called an iPod, a small pocket sized device capable of holding 1000 albums. If Apple had listened to users like me they would have built a faster, larger, better CD player. Instead they designed something I had no capability of thinking could even exist, I was thinking incrementally, Apple were thinking revolutionary.
You are better off understanding what users motivations are and then building something to satisfy that than involving them in user testing.