Using Qualtrics for Guerrilla Testing

The practicum group used a Qualtrics survey to gather quick feedback on two different interface mock-ups for the EDS project. With an afternoon of planning (mock-up creation and iterative testing of the survey with student participants) followed by 2 hours of guerilla testing in the lobby of Bobst, the team was able to collect over fifty responses from NYU students, faculty and staff on two potential layouts.

Qualtrics is powerful online survey creation and collection software used by NYU Data Services. Although Qualtrics can be used for a wide variety of applications, Qualtrics has a number of features that make it a valuable and flexible tool for guerilla user testing.

Skip Logic

Qualtrics surveys are created using blocks of questions. These blocks can be linked together using commands dictated by the survey flow. Skip logic, randomization and answer dictated question flows enable survey creators to shape the path of participants automatically. Skip logic was an incredibly important tool for the practicum team. Using skip logic, users who did not agree to the testing terms were able to be safely and anonymously lead to the end of the survey.

 A/B Testing

Using question blocks and the survey flow feature, the team was able to create a randomized ‘A/B Testing’ of both interfaces through a Qualtrics survey. Within a survey, users can be lead to a different block (in this case one distinct block was created for each section of questions related to a single interface) with a guaranteed randomization. Although A/B testing can be approximated by creating two separate surveys, these built in features from Qualtrics enabled testers to quickly disseminate the test without needing to keep a running tally of how many users had been asked to evaluate each interface.

Heat Maps for Mockups

Example of a Heat Map with Defined Regions

Using Qualtrics, the interaction designer’s in the group were able to quickly heat map two different interface mockups. After uploading a copy of the mockup into the software, the team was able to denote and name different regions of the interface. In the survey, users were instructed to select where they would start their research from if presented with the mockup as a working webpage, these ‘clicks’ were recorded as a part of a heat map (visually) and also as a part of a frequency table (collected as quantitative data) with each defined region as one of the table rows.

The benefits being able to present users with heatmaps in Qualtrics are the ability to test mockups (no need for a working prototype) and the region denotation. Defining each section of a page can help researchers to quickly synthesize and present testing finding with others with quantitative data.

Some Thoughts on Using Qualtircs for UX Testing with a Team

Overall the team had a positive experience using Qualtrics for guerrilla testing, however there are a aspects of Qualtrics that impact its use in user research and as software program utilized by a team. The most notable of issue that the UX group came across was with randomization and cache. As the tests were being administered in person with two laptops, the cache on the computers’ browsers would have to be cleared after each test was administered in order to ensure randomization in the A/B testing. As each session was registered separately, even when administered on the same computer without clearing the cache, this is an issue that was truly perplexing.

Another issue in testing was with image quality and size. When images were too large, it was possible for the image to cover up the instructions or buttons within the survey, however if the image was scaled in Qualtrics to fit, the image quality was greatly diminished. An easy work around for this issue is to always scale images with local applications before uploading them to Qualtrics.

Finally the group ran into some trouble using Qualtrics as a team. Opening, reviewing and editing a survey on multiple computers simultaneously caused parts of the survey to disappear and sharing the survey editing privileges with other team members took over an hour to transfer over. Based on these experiences the team suggests sharing a survey immediately between the entire group (upon creation) and working with local copies for editing, before pushing these changes to the final survey.


This post was originally published on the Pratt UX practicum at NYU Libraries blog here: 


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: