Evaluating Gesture Usability

[This is a rough draft/work in progress. Comments/feedback are welcome!]

There is some great work being done on how to design gestural interfaces but we don't see much work focused on how to evaluate their usability. Having reliable and repeatable evaluation techniques for gestures is important. Without them we can't tell developers if their system is getting better or compare two alternative implementations or products.

The following is a discussion of general issues to do with testing gestures for usability, followed by a description of specific test procedures and my experiences with them.

Gesture platforms

I've been working mostly with gestures on touchpads, so
that's my focus here. Many of the issues are the same for touchscreens and some are the same for free-form (in the air) gestures.

On a touchpad there's not necessarily a direct one-to-one mapping between your finger motions and the motion of things on the screen (in contrast to gestures on a large touchscreen, for example). I'm talking about gestures like pinch, swipe, and rotate, but also more abstract ones like drawing an infinity symbol to launch your favorite application.

Automated evaluation and learning algorithms

First off, I should say that I'm not talking about automated gesture testing with a dataset. Gestures are often implemented using machine learning algorithms. The developer can train and also evaluate the system using a very large dataset of gesture input that has either been recorded or simulated. The evaluation will produce percentages of correct and incorrect gesture recognition. This kind of automated testing should ideally be done as part of algorithm development and I'm assuming it's already been done.

You don't always need to implement gestures using learning algorithms. For simpler and smaller sets of gestures it might be easier to use heuristics in hand-written software. In that case it may still be useful to have prerecorded gesture input for evaluating the algorithms in a consistent way over time.

But I said I wasn't going to talk about automated evaluation… Assume the developers have done their job and you're a usability person asked to evaluate the gestures and give feedback. Maybe the developers have a dataset of gesture input data but they're not sharing it with you.

Evaluation scenarios
Here are some of the scenarios for when a usability test of gestures is useful:

  • Formative usability tests to help developers know if and how their algorithms are improving or not. These can also evaluate the gestures within a realistic context — evaluate whether the gestures work well in specified applications, whether the feedback and actions performed by the gestures are working appropriately.
  • Summative and competitive tests. Document the performance of the gestures for future reference. Compare the gestures on system A and system B, where the systems might have different hardware, firmware, or software, may be running different end-user applications, may have touchpads of different sizes or shapes, have different surface textures or different bezels. There are many variations you could compare.

What's different about gestures
Some aspects of gestures make them different from testing other user interfaces:

  • Gestures are inter-related. If your system has a single two-finger gesture, say pinch, then it's easier to make it work better than if it has two two-finger gestures, like pinch and rotate, or more. The power to discriminate between gestures gets smaller the more gestures you have in the same effective parameter space. This means that you can't test gestures in isolation.
  • Gesture interfaces typically don't have affordances. This means they must be taught. The user has to be prompted in some way to know to perform the gesture. Perhaps they looked at some documentation, watched a demo video, learned gestures on another system, or were just told that the machine had a certain gesture. The way people are instructed on gestures will have a major impact on how effectively they can use them.
  • Gestures don't just need to work — they need to not work when they're not supposed to. It should be difficult for a user to accidentally trigger a gesture.
  • Variation among users. Testing a variety of users isn't unusual, but for touch gestures things like finger size and fingernails can make a very big difference so it's important that the test participants are representative.

Research questions

main questions that gesture usability tests might try to answer include the

  1. Are gestures better than alternative (and existing) interaction techniques?
  2. How accurate is
    each gesture (on each platform)? What are the rates of correct and
    incorrect recognition? What are the failure cases like?
  3. How satisfying are
    the gestures to use? (and other aspects of subjective responses — how
    comfortable, tiring, responsive do they seem to users?)
  4. How effectively do
    people learn the gestures? How much, if any, training or documentation is
  5. What problems do
    people encounter with using the gestures? (development feedback)

Introducing gestures and testing the documentation

The documentation for gestures (or lack of it) may have a big impact on how well users can perform them. The documentation should itself be usability tested. Bring in users unfamiliar with the gestures and simulate the introduction that a typical customer might get to the gestures. Specifically, have them read the help text, watch the help video, or do neither. Then have them try each gesture. Record whether users can perform the gestures, or if not, how many need more instruction.

Part 2 is here.

2 thoughts on “Evaluating Gesture Usability

Leave a Comment