Three structured judgment evaluation techniques were applied to a system with a graphical direct manipulation style interface, to understand the types of usability problems they address. These evaluation techniques were cognitive walkthrough, heuristic evaluation, and the Smith and Mosier (1986) guidelines. The authors wanted to learn whether the techniques identify problems: across all stages of user activity, which noticeably affect users' performance with the system, and which are important to the usability of direct manipulation-style systems. Results showed that the cognitive walkthrough method identifies issues almost exclusively within the action specification stage, while guidelines covered more stages. The walkthrough was best, however, and the guidelines worst at predicting problems that cause users noticeable difficulty (as observed during a usability study). All the techniques could be improved in assessing semantic distance and addressing all stages on the evaluation side of the HCI activity cycle. To evaluate the directness of engagements, improved or new techniques are needed.
