Marcus Andrey Study
Ler em português Library

UX Research · Step 2 · 7 min read

Choosing a research method

There are dozens of research methods, and three questions narrow them down to two or three. The map NN/g uses (what people say or do, why or how many, and in what context), how the words of a research question point to a method, and why the best studies combine two.

A square board split into four cells. Back to front is what people say against what they do; left to right is why against how many, so the left cells hold a few big round pucks and the right cells many small square counters. Point at a cell and its pieces rise off the board. That's this step in one gesture: planning starts by choosing which kind of evidence the question needs. This step has five lessons: choosing a research method, writing a research plan, recruiting participants, screeners and sampling, and research data and privacy.Illustration: Marcus Andrey, drawn with Hairline (MIT, Lucas Marques).

The method comes last. The question picks it. Teams often start with the method they know: "Let's run a survey", "Let's do some interviews". Then they bend the question to fit. Turn it around. Once you know what you need to learn, three plain questions (do you need what people say or what they do? why, or how many? in real use, or in a set-up?) leave only two or three methods standing, and usually one obvious first step.

What you'll be able to do after reading:

  • Place common research methods on NN/g's map of the field
  • Pick a method from the words of your research question
  • Combine two methods so each covers the other's blind spot
  • Adjust the choice to real limits: time, budget, access and risk

1. Start from the research question

If you haven't got a research question yet, go back to Research questions that lead to decisions. Choosing a method without one is like choosing a lens before you know what you're photographing: you might get lucky, but you'll probably bring the wrong one.

With the question in hand, NN/g's framework places methods on three dimensions: attitudinal vs behavioural, qualitative vs quantitative, and the context of use (Rohrer, When to Use Which User-Experience Research Methods, NN/g, 2022).


2. The map

What people say, and why
  • Interviews
  • Diary studies
  • Focus groups
  • Participatory design
Needs, motives, mental models, language
What people do, and why
  • Usability tests
  • Field studies
  • Contextual inquiry
Where a design fails, and the workaround people use
What many people say
  • Surveys
  • Card sorting
  • Satisfaction scores
How common an attitude or need is
What many people do
  • Analytics
  • A/B tests
  • Usability benchmarks
  • Eye tracking
How often, how fast, which version wins
Common methods on two of the three axes NN/g uses: attitudinal or behavioural (what people say or do), and qualitative or quantitative (why, or how many). Top left, what people say and why: interviews, diary studies, focus groups, participatory design. Top right, what people do and why: usability tests, field studies, contextual inquiry. Bottom left, what many people say: surveys, card sorting, satisfaction scores. Bottom right, what many people do: analytics, A/B tests, usability benchmarks, eye tracking. Usability tests and field studies mix both kinds of data and sit near the middle. The third axis is context: natural use, scripted use, a limited version of the product, or no product at all.Diagram: Marcus Andrey, CC BY 4.0. Simplified from Christian Rohrer, When to Use Which User-Experience Research Methods (NN/g, 2022).

Say or do. Attitudinal methods capture what people say; behavioural ones, what people do. NN/g recommends that most usability studies "should rely on behavior", and notes that usability studies and field studies sit in between, mixing both, "though leaning toward the behavioral side is generally recommended" (same source).

Why or how many. Qualitative methods gather data "directly", by observing or hearing people, which lets you probe and adjust as you go. Quantitative methods gather it "indirectly, through a measurement or an instrument such as a survey or an analytics tool". That's why qualitative methods are "much better suited for answering questions about why or how to fix a problem", and quantitative ones at "how many and how much" (same source). Note what this means: five interviews can't tell you how common something is, and ten thousand analytics events can't tell you why.

Context. The third axis is how participants use the product during the study: natural use (field studies, analytics), scripted use (a usability test with tasks), limited use (a card sort, a concept sketch), or no product at all (an interview about their life). Natural use gives you realism with less control; scripted use gives you control with less realism.

The card Research method chooser puts the three questions on one screen, with a template for each research question.


3. Let the question pick

If the research question starts with…Start withWhy this oneThen check with
"How do people… today?" "Why do they…?"Interviews, or a field visitReal stories and context, before there's a designA survey, to see how common it is
"Can people… with this design?"A usability test, 5 per groupYou see the failure and hear the reasonAnalytics after launch
"What happens over days or weeks?"A diary studyMemory fades; diaries catch it close to the momentA follow-up interview
"How many people…?" "How often…?"Analytics, or a surveyCounts need numbers, and many peopleInterviews, to learn why
"Which version works better?"An A/B testReal behaviour, everything else held equalA usability test, to learn why it won
"How do people group or name these things?"A card sort, then a tree testTheir mental model of the content, then proof it worksA usability test of the navigation
The words at the start of a research question point to a method. The last column is the method from the other side of the matrix, which covers what the first one can't see.Diagram: Marcus Andrey, CC BY 4.0. After NN/g: Rohrer, When to Use Which UX Research Methods; Whitenton, Triangulation.

The opening words of a research question are a good hint:

The time dimension from What research is for adds one more filter: early on, generative methods; while designing, formative ones; after launch, summative ones.


4. Combine two methods

Every method has a blind spot. NN/g's answer is triangulation: "the practice of using multiple sources of data or multiple approaches to analyzing data, to enhance the credibility of a research study" (Whitenton, Triangulation, NN/g, 2021). The article is honest about why: small qualitative studies can't be statistically proven, and quantitative data lacks "context and meaning".

The most useful pairs cross the map diagonally:

  • Analytics + interviews: the numbers say where people drop off; the interviews say why.
  • Interviews + survey: the interviews find the themes; the survey measures how common they are.
  • Usability test + analytics: the test shows the failure; analytics shows how many people hit it.

You don't need both in the same week. Often the second method is data you already have.


5. Fit the method to real limits

The ideal method meets reality. Four limits shape the final choice:

  • Time. Interviews and usability tests can run in a week or two. Diary studies take weeks by design. A/B tests need enough traffic to reach a result.
  • Access. Can you reach the right people? Specialist and B2B users take longer; Recruiting participants covers how long.
  • Budget. Incentives, tools and, for in-person sessions, travel and rooms.
  • Risk to participants. A sensitive topic may call for a gentler method, like a diary instead of a face-to-face interview, or a lighter question.

Two choices sit inside every method:

  • Moderated or unmoderated. With a moderator you can follow up; without one you get more people for less effort, but the instructions must stand on their own. NN/g recommends pilot testing any unmoderated study, because "no one will be there to answer questions" (Schade, Pilot Testing, NN/g, 2015).
  • Remote or in person. Remote is cheaper and reaches more people; in person shows more of the context and the body language.

6. Common mistakes

  • Choosing the method first and bending the question to fit.
  • Using a survey to answer "why".
  • Using five interviews to answer "how many".
  • Running a focus group to test usability.
  • Never pairing methods, so every finding has the same blind spot.

7. Vocabulary

Term Meaning
Method A way of collecting data: an interview, a usability test, a survey
Triangulation Using several sources or methods so each covers another's limits
Moderated A researcher runs the session live and can follow up
Unmoderated Participants follow written instructions alone, usually through a tool
Context of use How the product is used in the study: natural, scripted, limited or not at all

8. Try it (no tools needed)

  1. Take a research question you wrote in Step 1.
  2. Answer the three map questions: say or do? why or how many? natural or scripted?
  3. Write the first method you'd use, and the second one from the opposite corner of the map.

Reflection questions: Which method does your team reach for by default? Which corner of the map has it never visited?


9. If you already practise

  • Review last year's studies on the map. If they cluster in one quadrant, your team has a blind spot shaped exactly like the empty ones.
  • Make triangulation cheap: know which analytics, support data and survey results already exist before each study.
  • When you present a finding, say which method it came from and what that method can't see.

Checklist

Learn more

Books, courses and videos to go deeper

Book links go to Amazon. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

What research is for · Writing a research plan · Card sorting and tree testing · Usability testing with five people · Contextual inquiry and diary studies · Analytics and A/B tests for designers

Saved only on this device.

↑ ↓ to move · Enter to open · Esc to close