10 Ways to Conduct Usability Tests with Credit Cards & Personal Data

A key principal of usability testing is that users should simulate actual usage as much as possible.

That means using realistic tasks that represent users’ most common goals on the website or app they’ll be working with ‘out in the wild.’

Usability testing is inherently contrived but we still want to provide as realistic a testing environment as possible.

For public facing websites this is usually pretty easy as the website is readily available.

However, if you need users to make a purchase with a credit card, register an account, login or need to use any other personal information, the testing logistics become increasingly complicated.

It’s difficult, and often prohibited, to ask users to use their own information. However, with a little creative thinking we’ve come up with some options that help us get good usability data while working within the constraints of purchasing, logging in, or working with users’ sensitive personal data.

Simulating Purchases

1. Have users stop before the purchase: If you’re looking to evaluate the carting process of a website, ask users to locate an item, add it to their cart and purchase all the way up till they click submit.   You can have them stop at the “buy” button if you don’t want them to fill out the checkout form.  This is by far the most common approach taken when testing eCommerce websites. Note: If users enter their own payment details, be prepared for some to accidentally click “buy” instead of stopping, so have ways of cancelling orders or reimbursing users.
2. Use a fake confirmation screen: Users get cagy when they have to provide personal information, even if it isn’t being submitted. The more personal, the more reluctant users become—from names and addresses to income, credit card numbers, drivers licenses and social security numbers. One way to offset this reluctance is provide a fake credit card number or account detail that will prevent users from making the purchase but allows them to progress all the way through checkout.

Making Purchases

Sometimes you need to test the entire checkout experience from finding to carting and to checkout.  This will allow you to examine confirmation screens and error messages. If you want users to actually make a purchase, you’ll first need to decide on whether you’ll want them to pick a product or if you’re going to pick one for them with a set price. The route you choose depends on the scenario you’re more interested in modeling and speaks to different stages of the customer journey (researching vs. ready to purchase).

Picking a Product: The advantages of picking a product are you can validate whether users select the right one, you’ll reduce browsing time and have more control over your budget. The major disadvantage is you’ll force users to purchase a product they may not want. An additional factor to watch out for if you pick a specific product is that it may sell out!  Even with just 20 participants, products can sell out quickly and create inconsistent experiences.

Setting a fixed price:  Users generally like being to pick their own product as it allows them to search for something they are interested in, creating a more natural checkout context. Things like colors, sizes, reviews, pictures and search results all matter more when users know they’ll be receiving a product. The disadvantage is that we’ve found many users will spend a lot of time browsing to find something they like.

We’ve found letting users know they have a set time limit (say 10 minutes) or let them know ahead of time to think of a type of product to search for helps.  Of course you don’t have to have a budget, especially if there are a limited number of products on the website, but it’s probably a good idea to keep budgets down and expectations set.

With a fixed price or fixed product approach determined, you’ll then want to decide if users will use their own payment method or one you provide.

3. Using provided prepaid cards: You can load up prepaid debit cards and provide these numbers to users when they make the purchase. The major advantage is that users don’t have to provide their own information and it will therefore be easier to recruit. You’ll need to be sure you have enough loaded on one card or multiple cards and that the bank security safeguards will allow multiple purchases to go through from the same website (often within minutes of each other).You can provide the billing information and users can then ship the product to their house or friend’s house. One unforeseen problem we had when using this approach was in one study, dozens of users actually used the billing information from the prepaid card as the shipping information. The card issuer then received dozens of products!
4. Have users purchase & reimburse them:  While gift cards will get users through the purchase process it won’t be as realistic as users using their own form of payment and billing information. Selecting payment methods (PayPal, credit, debit) and dealing with idiosyncratic shipping details are all common pain points in the checkout process.  Asking users to pay first will make recruiting more difficult and costly, but we’ve done it several times and include the reimbursement as part of the honorarium. We’ve done this both for our in-lab tests and unmoderated remote tests.  It’s easier to reassure people in-person that you’re not going to steal their information so you’ll need to add extra assurance when conducting an unmoderated study.
5. Cancel orders:  Instead of incurring the costs of reimbursing users you can cancel their orders if you have sufficient control over the inventory system, or if you can cancel them manually from emails or confirmation numbers. This can be difficult to do as most eCommerce websites have complex and quick automated fulfillment processes. When we tested rental car websites  we asked users not to submit the reservation. Due to a massive usability problem, dozens of users accidentally reserved a car! Because we had all users use a fake email address we were able to login to the account and manually cancel the reservations.
6. Have a subset of the sample purchase: A hybrid approach is often a good compromise for budgets and logistics. You can have a larger sample simulate the purchases and have a smaller sample (say 10-30 people) actually go through with the purchase. It’s fewer people to recruit and requires a smaller budget for reimbursing. You can use the same metrics as the larger sample (time, completion rates, perceived difficulty) then see if there are differences between simulated and actual purchasing. Our previous experiences with this approach indicate some measures remain consistent whereas others diverged. In one study, users spent a lot more time browsing when they had to purchase a product versus locating one they didn’t get to keep.

Mockups & Demos

Sometimes you can’t use the live system because of security reasons such as users needing their own accounts (banks or credit cards). In such cases you might consider mocking up part or the whole system or using a test or demo version.

7. Use a demo or test system: For credit card systems, bank accounts and other systems that require an account, there’s often a demo or test version. Demo’s usually contain canned data so there are limits on errors and some functionality but it’s surprising how even these fake systems provide valuable insight on the navigation, form fields and flow. Test systems are often staging “sand-boxes” for the developers to ensure fixes and features are working. Test systems usually allow users to encounter real errors and messages, realistic load times and full functionality with fake account credentials. Watch out for unanticipated downtime and bizarre errors—both of which are the price to pay for using the test environment.
8. Use a dummy account: In both live systems and test systems there’s often a dummy account that the developers create for testing. It can be either a dummy login or dummy credit card which will simulate all functionality. If it allows for concurrent logins and access to full functionality it becomes another alternative.
9. Mockup the whole flow: If you need to test a competitor system that requires an account and you can’t have dozens of users accessing simultaneously you might consider mocking up a high fidelity version using software like Axure. You can use screen shots as background and just enable form fields and navigation to get a good representation of the experience. Keep in mind that this approach won’t simulate load times or error messages that can have a significant impact on the user experience.
10. Mockup some screens: Often mocking up just part of a task or set of screens will accomplish your goal. This may work if users stop before purchasing and you want to test the confirmation screen. Split the experience up across tasks, have the users pretend to make the purchase on one task then show the mocked-up confirmations screen for the next task. You can also do this using prototyping systems, screenshots or some simple HTML with hotspots.