Your boss walks in and says your software/web project needs to be tested with real users, and he wants the results yesterday. Suddenly, you’re faced with a new problem outside your comfort zone: You need to validate your project with a lot of people, do it quickly, and the results could affect the entire direction of the project. So what do you do?

Agile Repeated Testing for Enhancement is a quick way to get the answers you need to ensure your project lands well with your customers. Here is a 5 step guide to help you through the process. 

What is ARTE?

A: Agile. You’ve got to do this quick, but clean. Cut through the red-tape, and get things done.

R: Repeated. You’re not going to just talk to 1 person. You’ve got to make this iterative so you can get as much data possible in your limited timeframe.

T: Testing. Design like you know you’re right. Test like you know you’re wrong.

E: Enhancement. Tune everything until it’s optimal. Don’t cut corners, but streamline what should be done, and reduce or remove things less critical.

Step 1: Kickoff & Knowledge Gathering

Whether this is your project, or you’ve been brought in as an outside consultant, you need to have all the right information to get started. Try answering these questions to curate your ARTE:

• What is the business application you’ll be testing? Be specific here.

• Is it an entire package, or just a few features?

• If it’s available on multiple platforms, will you be differentiating between the types of platforms available?

• What kind of data do you need to tell the final story, and what type of data do the executives think they need? (More on helping people understand what kind of data they need to answer their business questions later). 

Step 2: Participants—Who, Where, How Many

This is critical. You must identify your target population well to get the right data. If you're testing an in-dash navigation system, you probably don’t want a user who bikes to work every day with no intention of ever buying a car. If you're optimizing an app for iOS, you may or may not want android users. It all depends on the goal of the test. Clearly think through who the users will be, where you will recruit them from, and how many participants you need to achieve reliable data. We won’t discuss sample size here, but it’s something you may need to take into consideration. There are times when the sample size is determined more by how long you have to conduct your test, and less by achieving a statistically significant sample size. See the diagram below to better understand how to determine a testing schedule. Ultimately, some are better than none. Generally.

The Math:

1. 40 participants

2. 2 days of testing

3. 480 minutes (8 hours) per day

4. 15-minute gaps between participants

((Num_Minutes * Num_Days / Num_Participants) - Gap = How much time you have to work with each participant

or

((480 * 2) / 40) - 15 = ~ 9 minutes per test (per test participant)

Step 3: Script Writing

It’s great that you know who you're going to talk to now, but what are you going to say? “Hey do you like this feature?” is probably not going to be sufficient. Well-crafted questions are both art and science. There are times when a yes/no response is appropriate, but there are often times where you want to include open-ended questions. Open-ended questions are questions specifically designed for an explanation instead of a yes/no response. They allow you to dig a little deeper into how the participant feels about a feature and give you greater insight into the product you're testing. These are great for your qualitative report. For the quantitative aspect of your report, you will need a categorical variable. A preferred scale to use is a 5 point Likert scale. The participant can rank a very large range of things such as sentiment, emotion, agreement, frequency, importance, likelihood and more on a simple Likert scale. 

The script is more than just the testing questions though. It also includes an introduction, scenarios, and any important disclosures. Are you recording the session? You can include the consent process in the script, or simply a question to confirm the consent is on hand.

Step 4: Logistics

There are a few additional details that you must work out before you start testing. Where will the testing be conducted? Is it a space you own, or a public space? Do you need permission to test there? You need a location that has good wayfinding so your participants don’t get frustrated trying to find the correct location. You will likely need a waiting area, a room for testing with minimal distractions, and appropriate staffing to keep the flow going. The same goes for virtual tests hosted on platforms like Zoom. Getting people to the right room, properly set up, prepared to share screens or devices is no easy task with many participants, and may require additional coaching and written instructions before and/or during the meeting. You need to account for those tech hurdles in your schedule.

You also need to consider what software and equipment you’ll use. Will you be recording audio and video? If so, how? Will others need to be observing remotely? Do you have backup equipment if something doesn’t work properly? Yes, equipment failure and computer crashes can happen in the middle of a testing session. Be over-prepared. 

Step 5: Practice & Revisions

So now you’re all ready to go, right? Wrong. Now you need to do a few dry runs. First with your team, and then with someone outside your team. Test the questions. Are they leading? Are they repetitive? How is your timing? If you have participants coming every 30 minutes and your script is taking 40-45 minutes, you still have a lot of work to do. Get the script right. It is well worth it.

Finally, be prepared for it all to go wrong. We’ve seen people show up late. Show up 2 hours early, even show up 24 hours early! We’ve seen outdoor tests get rained on. We’ve seen on-location tests where no customers showed up. We’ve seen intercept interviews where nobody would talk to us. You never know what’s going to happen, so over-prepare, then be ready to pivot. When the timeline is tight, fortune will favor the prepared

Great, now you know the steps to curate your ARTE session. If you need help with any of your usability testing and design needs, please reach out to us at Precocity. Our team of usability researchers have a passion for improving the customer experience on a variety of platforms.