This application works best with JavaScript enabled.
Ilustrační obrázek s textem -

How to use remote user testing

  • ux
  • cx
  • user testing
  • remote user testing
We want to share a few tips with you on how to carry out remote user testing. We have gained a great deal of experience with it during the time of the “new normal” and we will be happy to help make testing work for you as well.

The prerequisite, just like user testing in the lab (or meeting room, or other “testing room”), is that you already have and know:

  • the goal of testing,
  • hypotheses,
  • testing scenarios,
  • subject of testing (i.e. functional website, application, prototype, or only sketches – yes, even these can be placed into electronic form and used in remote user testing),
  • respondents gathered from the corresponding target audience (or you at least have your target audience of respondents defined).

A necessary condition for everything to proceed successfully and for you to enjoy the testing process is – and in this sense, remote testing does not differ much from presentational testing – quality preparation. This text is a summary of my personal experiences and knowledge in the role of moderator, the experiences of other moderator colleagues, and also of the participants (respondents) in remote testing themselves, with whom I conducted a small survey before writing this article.

What concerns moderators about remote testing, and how to fix it

We won't have enough time.

Solution: Do your remote pilot testing with a friend or colleague as a live test, so you can uncover everything that will need to be fine-tuned (modify the script, prototype, planned time allocation, method of acquiring recordings or minutes, functionality of all equipment, etc.).

We don’t have a perfectly prepared prototype.

Solution: Not a problem. Prepare your script in such a way that it reflects the current status of the prototype, and stick to it.

I don’t know what kind of connection the respondents have, and I’m afraid the call will drop.

Solution: Plan and carry out test calls (1-2 days before actual testing for about 15 minutes). During the test calls, inform your respondents about the process of the testing itself. Make sure that the call is taking place from the room where the testing itself will take place, and at the same time, so that you can test the actual connection and specific equipment. Test the picture and the sound, or the use of headphones if applicable. Send the respondent e.g. using the chat feature of your online tool a link to a practice prototype or any web page and make sure he or she will be able to screen share it. Respond to any questions and confirm the agreed date of the test. Agree on a communication method if any problem occurs regarding participation or connection on D-Day.

I don’t know whether the respondents will be completely focused on the testing or whether they will be interrupted (by their daughter studying at the table next to them).

Solution: During the test call with each respondent, agree to arranging the necessary conditions for smooth testing. Any impending disruptive elements must be addressed in advance, otherwise it will be better to agree to a more suitable data in order that the planned time is not extended and that it doesn't influence the scheduled time for the next respondent. Even so, count on a time buffer between individual respondents (30-60 minutes).

What concerns respondents about remote testing, and how to fix it #

It will be complicated.

Solution: Explain respondents that you are never testing their abilities. You are testing the project or service. The goal is indeed to discover all complications. So if something proves to be complicated, that’s a great benefit.

The connection will stop working or they won’t be able to handle the technology.

Solution: Reassure the respondents that most problems can be revealed and resolved during the test interview. If the technology or connection fails.

What respondents actually enjoy and can motivate them

  • The feeling that you are part of something new or that you are helping or improving something; which they might ultimately end up using themselves (they should be part of the target audience).
  • Chance for new experiences. Opportunity to learn something new (technology, online environment).
  • Communication with new people outside your own bubble.
  • Opportunity to give your opinion, which will be taken into consideration in the creation of new products.

Advantages and disadvantages of remote user testing

Advantages

  • Relevant and honest feedback is easier for the respondent, and the entire progress of the testing is smoother, when the user is in their natural environment where they feel safe.
  • Option to connect with anyone, anywhere (throughout your own country or the world).
  • Saving on travel and the costs associated with it.
  • Saving on organization and communication with other people involved, e.g. at reception (instructions related to the test venue, parking reservations, etc.)
  • Even immobile respondents can participate.
  • Lower time demand enables participation of a larger number of respondents; e.g. mothers on maternity leave (they can easily participate when their child is sleeping)

Disadvantages

  • Lack of personal contact that allows you to perceive all important emotions.
  • You are maximally dependent on the quality of the connection (including video and audio quality).
  • It is much more demanding to retain full attention and concentration on both sides.
  • Respondents have total control over the test location, i.e. preventing distractions is dependent on their responsible approach.

Tips and recommendations for the organization, progress, and technical solutions, or what not to underestimate

Suitable environment

Without distractions both on the part of the moderator and on the part of the respondent.

Scope of testing

Test more often and in small amounts. Feel free to use only one monitor you are comfortable with for all interactions.

Technology and connection

Including recording.

Pilot test with a volunteer

To test the time demand and any need for editing the script or the subject of testing on the basis of problems determined.

Participation of a note-taker

In ideal cases.

Test interview with respondents, whose purpose is to:

  • get to know one another,
  • ideally address written consent to recording the interview,
  • outline what will take place on D-Day,
  • test connection and equipment (including audio and video),
  • testing work with online tools (chat, sharing images, etc.),
  • relieve the respondent’s fears of the unknown = work together to fine-tune any possible problem areas,
  • if applicable, the necessary settings in the respondent device (e.g. activating “developer mode” on telephone in order to visualize taps on the screen).

Data volume of prototype

It is necessary to count on a worse respondent connection, the need to share screens, and the recording, all via videoconference.

Consent with recording

Should be formally resolved even prior to the actual testing in order to avoid unpleasant surprises and subsequent problems when carrying out the testing.

Repeating process agreed at test call

Such as sharing, using chat, etc.

Duration of testing

Ideally keep it within 1 hour; anything else tends to be harder on the concentration both of the respondent and the moderator.

Time plan

Schedule all respondents with sufficient breaks and buffers in the event of any unexpected delays. When scheduling, be mindful of any time difference for respondents due to their time zone.

Mutual contact info (tel. number)

In the event of unexpected problems with participation, connection, etc.

Introduction

It is worth taking the time to establish an informal atmosphere that will help the respondent warm up.

Good name

If you make any changes to the respondent’s device settings (e.g. activating developer mode on the mobile so that “tapping” is visible during testing), don't forget to help the respondent return the device to its original state.

Testing online tools

Skype (testing on mobile)

  • Expanded use in companies and privately.
  • Need to create an account (registration), where each user has their own nickname.
  • Allows sharing, chatting, and recording (no need for external solution).
  • After completing recording, the recording is sent via chat (note: here it is a good idea to download the recording locally and delete it from chat; the downloading takes a while and before you delete it from the chat, you need to make sure it is fully downloaded).

Google Meet (testing on desktop)

  • Expanded use in companies and privately.
  • Does not require registration of every user or special installation – for connecting to the meeting, all you need is to click on the link provided.
  • Allows sharing, chatting, and recording (no need for external solution).
  • After the recording is complete, the recording of the call arrives automatically along with a transcription of the chat communication to the Gmail of all invitees + is saved online on Drive and shared for all invitees (note: the organizer of the meeting has the option to download the recording).

Whereby (testing on desktop)

  • Expanded use in companies and privately.
  • Does not require registration of every user or special installation – for connecting to the meeting, all you need is to click on the link provided.
  • Allows sharing and chatting free of charge (recording must be handled externally, e.g. on a Mac via the native application QuickTime Player)
  • The paid version enables recording (no need to handle it externally).

Closing bonus

Remote user testing has “boomed” mainly due to the COVID-19 situation and I believe that it has been eye-opening for many of us (like me). I see it as a shift out of the comfort zone and that – I hope you agree – is never a bad thing! It has definitely expanded the opportunities for moderators of user testing. So once the world will be “normal” again, it will be easier for us in that we will be able to choose whether it is more appropriate to do our user testing “in the lab” or remotely.

So? Are you looking forward to doing your next user test remotely? For us in the Lundegaard design team, such testing is a standard tool. We are happy to share our experiences in greater detail. If you have any questions or need advice or help getting it done, don't hesitate to contact us.

And a closing tip:

if you don't have the respondents you need and you don't know where to get them, we recommend you get in touch with Zuzka Štědrá), who can sometimes perform miracles.

Are you interested in working with us?

Send us your contact details. We will be happy to get back to you.

Contact us