How to Conduct a Usability test on a Mobile Device
Jeff Sauro • May 1, 2012
When it comes to testing websites there are many unmoderated and moderated solutions.
But if you've ever tried to evaluate an app or website on a mobile phone or tablet there are fewer options.
Usertesting.com offers a new mobile testing service
which recruits users and records their mobile devices while they interact with your app or website.
Tools like Userzoom
also collect some data from users on their devices.
But for an in-person moderated usability test on a mobile device, here's the setup that we've found works well.
What you'll need:
- The MOD 1000 or other macro-focused camera
- Facilitator Computer
- Screen Recorder : GoTo Meeting, Camtasia or CamStudio
- Computer to administer tasks to the user
- External Microphone
- Webcam (Optional)
We have our usability lab positioned with the facilitator and user facing each other in a perpendicular arrangement (see the diagram below). This allows for easy face-to-face communication and the occasional intervention when the user has problems with their computer or device.
Recording the Screen
Let me save you some research. As of spring 2012 there isn't a way to record the real-time output of an iPhone or Andoid device without rooting or jail breaking the device (something most users won't want you to do to their phone). There isn't a Camtasia or GoTo Meeting solution for mobile devices that allows you to share and record the screen.
However, even when the technology becomes readily available like on desktops, it probably wouldn't be sufficient. When we record the screen of someone's computer we can see the mouse movements and keystrokes so a simple output of the monitor captures almost all the relevant interactions.
On a mobile phone it's the fingers that do the talking instead of the mouse pointer. What's more, mobile phones are well, mobile, so users will move and rotate phones all the time. You'll want to capture as much of that as possible. This means you'll need an external camera.
We've tested many users and tried many setups with different combinations of cameras and rigs. The one that works the best is a camera and rig we developed ourselves—the MOD 1000
The MOD 1000
evice) is our lightweight rig for recording mobile screens and interactions. The macro focused camera is mounted on a lightweight aluminum plate with a grip-tight surface. The image of the user's phone stays still while the user moves and manipulates their device instead of being confined to laying it flat on a desk.
The software works well on Macs and PCs. The MOD 1000 allows for landscape or portrait displays and works on all size smart phones. The cradle is light and our users seem to stop noticing they're holding it after a few minutes. The camera on the MOD 1000 plugs right into the facilitator's computer, so the facilitator can see exactly what the user is doing in high resolution.
Displaying Task Scenarios
We use UserZoom to display task-scenarios in randomized order and collect post-task/post-test questions along with task-times. We create a background screen that has the most important global information for the user throughout the test--such as WiFi username and password and the test URL. Often we are testing both a client's mobile and desktop interface so having the tasks in a tool like UserZoom allows us to more easily collect data on the same tasks for both mobile and desktop interfaces.
Sharing the Screen
We have found that it is necessary to share the screen from the participant's computer with the facilitator's computer. We use Join Me
, a free lightweight web application that allows for screen and mouse sharing. Just download the app on the user's computer and start a screen share. On the facilitator's computer, go to join.me in your web browser, type in the code from the user's computer, and voila, you can see exactly what the user sees. You are then able to take control of the user's mouse as needed.
We like having control over the user's computer as we find many users forget to start the tasks as they are usually focused on using their mobile phone. When the facilitator has control of the participant's screen, the facilitator can go ahead and start the tasks if the user forgets and advance the screen as necessary. Seeing the user's screen is also helpful for knowing what task users are on when reviewing video recordings.
We find it helpful to have a webcam pointed at the users face and phone with the feed going right to the facilitator's computer. This second camera provides an alternate view which can be helpful for capturing how the user is physically interacting with the device (more so than say a stationary computer) as well as getting those informative facial expressions of frustration or delight.
We record our sessions and often have clients viewing remotely or from the observation room so having good sound is important. Because the user and the facilitator are not sitting around one computer, the built in computer microphone will probably not work all that well. We have found audio from a good webcam provides clear sound from the user and facilitator or alternatively from a good external microphone (there are many bad ones out there so try a few).
Recording the Session
We use GoTo Meeting to both share and record the sessions. We often have clients viewing remotely already, so using GoTo meeting's built-in recording works fine. GoTo Meeting can save the audio and video of the sessions as a compressed and sharable .WMV file that can be used for creating highlight reels and reviewing the interactions later.
After each session, it takes GoTo meeting about 30 minutes to compress an hour of video. GoTo meeting doesn't save the web-cam of the user, so to capture the user's screen, mobile phone and desktop we use Camtasia
As mobile applications and websites continue to proliferate the need to quickly test and evaluate will also grow. For now, moderated "lab" usability testing is still one of the best ways to capture the rich experience of interacting with a mobile device.
We hope this quick overview of our setup will help others gather the data to continue improving the mobile user experience...we can then continue debating the best way to display mobile websites