July 31, 2019 | Day 3, Session 2

Research is sometimes a bit of a neglected flower in a lot of companies. So how many of us believe that user research is valuable? 10 years ago it was not that many hands.

How many of your teams actually do user research on a regular cadence? Less

How many of you feel your research makes the impact you want? Far less hands.

Most people don’t feel that the research they do makes the impact they want it to.

We know how to get research done:

• talk to humans
• ethically obtain their consent
• provide them with something to look at
• collect responses
• offer them some compensation for their time
• bring back evidence
• take care of them
• take care of ourselves
• be good citizen

We can do all of this really well now.

But what do we need to make research matter?
What do we need to do to make research COUNT? We need:

• relationships throughout the company, allies
• budget (maybe)
• participation in research in the company
• tangible outcomes – come back with results we can show will have an impact on the product or service we’re designing

Adam Thomas- product manager in NY
“I hate writing the quarterly research report but love how it looks when the leadership teams eyes light up. A lot of work but so worth it if you want research to be taken seriously in your org.” (twitter)

If you are doing:

• real connected research on a regular cadence
• and are sharing it with people who matter
• and they’re excited to hear it
• because they know it helps them

…then you’ve won!

What really matters is doing real research and people know it helps them.

When does research really count? It counts when it answers a question that people in the organization care about.

So what do people want answers? Can you become the source of answers to questions that other find difficult to solve?

To become that source you have to know what people need – who is likely to need something, they have to know that you’re able and interested.

You could be the person to answer about things like:

• oddities in analytics
• weird feedback from customers
• unexpected uses of products
• strange hunches (not just your own)

If you can do qualitative research you’re the one that can answer these things.

It all ties back to curiosity. But we need people with some power in an org. But she doesn’t necessarily mean hierarchical executive power.

Curious people with power are the most useful ones to influence.

A lot of companies want to give engineers what they need, so if you can find an engineer who is curious about research you may want to answer their questions!

One of the most interesting things they found at 18F when they researched digital transformation was that people in mid-senior positions are incredibly important for influence.

You have to open up your research practice. It can’t be something that happens only inside your department and you can’t talk about it in weird jargon.

• don’t mystify about the work
• put as much effort into communicating as into practicing
• create opportunities for people to ask questions and learn

We’re at the point where most people want to do this but not many people know how.

So how do you bring people inside a research practice?

• hold office hours
• open a Slack channel for questions / discussions about research
• brown bag program – when people understand how something works they’re more likely to support it
• host research sessions where people can sit in the back and eat their lunch while observing research
• have a newsletter
• wall work – she had done a project where they had 75-125 website

RESEARCH COUNTS WHEN PEOPLE CAN SEE IT

…with their own eyes

One of the things she sees when she works with mission orgs is that people joined a career because they want to have a particular impact out in the world. But in an org where research isn’t present it’s really easy to lose contact with the impact we’re making. This can be discouraging.

People do better work when they see the impact of the work they’re doing!

Who in your organization is disconnected form their user?

“When someone has been in the field with you, the data doesn’t have to be explained.” – Meena T.

INVITE PEOPLE ALONG

• whoever is curious
• whoever is disconnected
• whoever is influential (even if they won’t come) – invite you stakeholders into a more colorful world
• show them what you’re doing
• make the rules clear and set expectations
• send them home with gifts

What is included in an invitation?

• who is it from?
• what is the event (what will happen there?)
• when and where is it?
• what will be provided?
• what do I need to bring? dress code?

You can invite people via email, a bulletin board, a text, whatever works within your org.

Basically you’re Willy Wonka here. You’re saying come see this world you’ve never seen before. You want to design the experience when inviting people along to research. She sometimes makes handouts or does demos. Make sure everyone has the tools to enjoy the experience. Giving everyone a responsibility is also a good idea, like different roles in taking notes like “I’d love it if you would not all of the time someone stumbles on a legal term.” This will engage someone that’s not actually a researcher into the process of doing it.

People Who Get To Ask a Question Buy Into The Results

In her experience, a client who got to ask a question of a participant was 100% supportive of a study!

Mix Your Research Methods

• interviews
• diary methods
• card sorting
• shadowing
• grounding the user research into other types of research can create a clarity – connect it to other types of data people are used to reading.

Ex: 2008 a company was researching what people were doing with phones in their cars. They did the following:

• ride-along interviews
• broadcast to 2 continents
• observed
• took photographs and grouped those
• video editing

Later in 2016, she got asked: What are the best practices for digital transformation?
How do you even answer that question? What does that mean? What do we do?

Method hypothesis:

• Interviews??

They realized they had to refine the question. If you get assigned a question that is confusing or vague, it is ok to say ’that is not answerable.’ Unpack every single one of those terms until you can unpack a question that you can actually devise a research method to answer.

The question they got to was: What makes modern digital practices stick within a government entity, beyond a single innovation project?

So then they ended up with these questions:

• what does digital transformation mean to you?
• How do yo know when you’re doing?
• what are the biggest obstacles to this work?
• how do you make the changes last?
• Who else in the organization had an important role in this moving forward or not?
• what resources would be valuable for other agencies doing this work?
• Then they tried to recruit 3 people at different levels of the org who were around for the same time at the time of the project.

They did moderated interviews with 20-25 people, they did a literature reviews, a policy & regulation review, did cluster recruiting (mentioned above, which required really tight consent on what they were going to do), tight consent & confidentiality. They keep transcripts in a special drive, disassociated the names of the people from transcripts entirely, changed the names of states, etc. This mixed-method study ended up being so much more valuable than just a set of interviews.

Book: Mixed Methods by Sam Ladner

Doing it Wrong

This is a really valuable skill to have!
If you cut the right corners at the right time you can be the fastest and cheapest resource in the company.

She can go out and do a half-day study with just the below:

• laptop
• notebook
• content forms
• phone to take photos
• $20 to buy people coffee

You can also set up a script & score sheet, label your assumptions on it, and that can be a great tool for a quick and dirty study!

This is a half sheet where you can collate results of a research study!

After every interview or shadow, wright down:

• 1 sentence description of the person (tall AA guy in a nice suit, friendly. white woman with 2 children in a stroller and a big bag) – this helps you remember the person
• #1 thing that surprised you – it’s rare to have a research conversation without a surprise and this is often the key
• quick summary of 2-3 things they said (can be your scoresheet)

Research Counts When We Can Tell The Story Really Well

Research is our product and the rest of our organization are our users – how are they going to receive the story of the study?

Synthesis works best as a conversational practice.

The first time you’ll get the stories out is right after the research process. All of the people involved need to participate. It’s important to think of yourself as a facilitator of this session.

We want to state our goals, but we’re much more there as equals. We’re sharing everything we all collectively captured from our interactions with humans out in the wild.

“In order to understand what another person is saying, you must assume that it is true, and try to imagine what it could be true of.” – Miller’s Law

Ask why is this so important that someone is having such a strong reaction (if that’s the case) and what I can learn from about this?

If someone uses a metaphor that is extremely helpful.

If people talk about something in 5 ways it’s virtually certain at least one of them will be an apt metaphor.

So she’s added “can you say that another way?” to her question set. She uses this with interviewees but also the team. We are working together to exteriorize thoughts and turn them into language, then specs and products and real things in the world we’re trying to design.

“Spend as much time on communicating outcomes as you did on executing the work.” – Paul André, Facebook

REMEMBER TRACKING? HERE’S HOW TO DO IT!

After every day of research, by email or Slack:

• we talked with X customers
• the most interesting thing we saw/heard was..
• themes that are emerging or fading
• how we’re pursuing this tomorrow

After each study:

• we talked with X customers
• here is the question we were trying to answer
• this is the story of what we learned
• here’s what we need to do

Ways to tell a story / ways to communicate the results of a study:
If she does research and records sessions and people get so frustrated they say a bad word, she edits together a video of all people

* wall work
* giant PDF reports (sometimes these really win)
* edited videos
* backlog tickets
* design artifacts
* prototypes

The Government Digital Service in London uses walls to display research work. It’s colorful, bright, very clear in a lot of cases, but you can’t walk around their office without knowing what the research team is finding out about ‘what’s it like to move from somewhere else from London? what’s it like to be a parent of a small child in the UK?’

The single-best case you can make for research is an illuminating story.
Sometimes one story that is real can reach people and be the foundation of a project.

Sometimes you do have to fight (nicely).

Why don’t people want to do research?
* it takes too long
* agile doesn’t fit design processes very well

You can do a really solid research study in 3 days:
* Day 1: create the thing you’re going to test, write scripts & scoresheets, buy incentives (usually can be small)
* Day 2: go where the users are and intercept them
* Day 3 morning: facilitate a synthesis session with everyone who participated in research
* Day 3 afternoon: host a show-and-tell

She believes most people can get ahold of this much time and get this first set of research done that will help get a compelling story to help people understand why it matters.

What if you can’t talk to customers? Like what if you’re not allowed?
* focus on your company’s sector more generally
* what groups of employees can stand in?
* where do people who need your product go?
* get topic-related stickers and business cards if you don’t want to reveal who you work for
* consider using personal social media – it is something you can consider as a patch

Research participants – reliability of participants from unreliable to reliable) :
* team members (don’t use them as participants)
* friends who are not on your team (if you acknowledge the pitfalls)
* target customers are pretty good
* real users with recent experience is pretty good
* real users with current needs is good
* real users with a need THIS HOUR is the best thing you can do.

What is the exact ROI of a research study in dollars?
It’s hard to calculate, but she will say if you can get your hands on $200 you can do valuable research.
* $60 incentives (6 $10 starbucks cards)
* $40 equipment (clipboards, backup drives)
* $30 stickers or Moo cars
* $70 parking, taxis, meals, etc.

You can do a 1 day study with this kind of budget.

If you are small and cheap people stop asking about the specific ROI and start looking toward the positive value of what you did. If you came back with a compelling story they’d probably have been happy for you to have spent $1000 even if you didn’t get approval up front.

Remember: you can explain
* oddities in analytics
* weird feedback from customers
* unexpected uses of products
* strange hunches (not just your own)

The instinct pieces, the emotional pieces
Research stands on its own when you can lean on the things statistics can’t do alone.
Behavioral patterns repeat over much smaller samples than emotional patterns.

Between 5-20 for behavioral research is enough to find what you’re looking for.

An intern does not have the organizational power to deliver the findings, even though they can execute the research.

If research is about
* making space for participating
* raining questions
* leading conversations

These are senior skills so it’s not fair to ask an intern to do this.

You can ALWAYS do some kind of research.

But what if no one seems interested in research? It’s not really true. There is someone in customer service who has closer connection to users that wants to bring light to the stories they’ve heard.

Put questions up on a wall.
* Make a pain-point video and ship it to executives.
* Make a little bit of good trouble in the organization.

You can get through the no budget program with a $200 research plan.
* Patch the gaps with personal equipment.
* Go without incentives
* Try to get approval to use swag & business cards.

the most effective thing in changing practice is an army of friendly people, who can sit next to the people with older practices and show them the new practice without judgement.

That is why the dandelion is the poster-flower of this presentation.

Research is something that needs to cast seeds really widely, and it’s something that once it gets out there is hard to get out of an organization.