The validity of Usability Testing, thought by many to be infallible at detecting a products defects, is called into question by industry leaders.

In a panel discussion at CHI2003 it was discussed whether a) the majority of defects in an application are found in the first 5 subjects as espoused by many Usability pundits, and 2) Does the same test run with a different sample reveal the same defects. The answer seems to be no.

A strong case was made for the “Usability By Design” concept where testing fine tunes and validates design and is not used to shore up poor design. If a system was well researched in the beginning and created through iterative design with user participation, then there are less likely to be these surprises later on.

Jared Spool shared his experience where new problems continued turning up even after the 16 participants were run through the test. Similarly, Rolf Molich ran a set of tests in replication where 310 usability problems unearthed, 75% of which were not overlapping.

What could be going on here?

First off, one could say that if there were 300+ problems in a system than it wasn’t ready for testing. It should have still been in the iterative design cycle or at most undergoing expert review.

Second, it would be important to know the methodology of the tests in question. Many times I’ve seen tests being run in the guise of “Usability Tests” which were little more than a direct observation session with a vague set of tasks to fulfill. Deborah Mayhew covered thus problem in Usability Testing: You Get What You Pay For, though I’m sure this was not the problem with the studies quoted in this panel.

The primary responsibility of a researcher is to conduct a study which has both internal and external validity. Internal validity is created be controlling all aspects of the study and manipulating the independent variable to see what turns up in the dependent variable. Let too many slips happen in your control and the studies validity falls apart.
Usability News: CHI 2003 Feature: Testing… 1 2 3 4 5 … Testing…