Lessons From The Trenches Of UX Research: An Interview With Brooke Baldwin
In this exclusive interview with VWO, Brooke Baldwin takes center stage to share her experiences, anecdotes, and learnings.
This article is a transcription of the ninth session from VWO’s Masters of Conversion webinar series featuring Brooke Baldwin, UX Research Lead at WhatsApp Inc., and Aniruddh Jain from the VWO Marketing team.
In this episode, Brooke talks about her passion for UX research, what her professional journey of more than two decades has looked like, and what these years have taught her about the human psyche. VWO learned a ton from her and thought that it was prudent to summarize the key highlights and insights from the conversation:
Brooke’s professional journey and her role at WhatsApp
Brooke started working with startups while she was in graduate school and has been in product design and research for a little over 20 years. While she initially had no master plan, she took up every opportunity that allowed her to learn and grow by working with people having diverse skill sets which amplified her innate skills along with helping her borrow skills outside of her T.
A few years ago, she took up an exciting role at Facebook, where she worked for their Ads- Emerging Markets team in London. It allowed her to travel all over the world, spend time and conduct research in emerging markets including India, Mexico, Brazil, and Indonesia, and learn more about countries she’s always admired.
About a year and a half later, she thought it was time to challenge herself and look for other exciting opportunities within the company. WhatsApp had just decided that introducing a User Research team made business sense, and she was offered the chance to be one of the first researchers in the team. She thought it was the perfect next step for her given WhatsApp is a huge product in countries she’s always loved, been interested in, maybe have lived in, or have spent quite a bit of time in.
What keeps Brooke motivated and excited about her work?
Brooke likes to learn new stuff and would get bored if she has to do the same thing over and over again. She feels UX Research is a job that continually challenges her and offers an immense amount of learning till date. After all, it’s a job tailored for curious souls who are professional tinkerers.
She adds that the experience of witnessing engineers, product managers, or data scientists from her team get their hands dirty in the field with her and see how users in downtown USA or Hong Kong perceive the utility of the countless WhatsApp features is an education above par.
UX Research for a Billion people
Brooke highlights that as a company, WhatsApp never tries to move the fastest to get the shiny new feature out. With over 2 Billion monthly active users, it is imperative that they take their time before any release to ensure that the voices of users from key markets and countries are heard. Therefore, UX research, in WhatsApp’s case, typically involves much more than just traveling to one country for a week and talking to 20 users, for instance. She also talks about the fact that WhatsApp’s usage could vary drastically across geographies and analyzing the differences and similarities among such a diverse user population is a very important part of her job.
Evolution of UX Research over the decades
Brooke recollects that when she first started out, her role was titled ‘Information Architect’, and usability testing was one of the foremost and primary responsibilities. Usability Study was one of the first things she learned where she’d expose a group of people to a prototype or simply a design on a piece of paper and have them work their way through it, which would help her figure out whether or not the design worked.
Over time, she feels that the umbrella of UX has evolved to embrace many disciplines (such as psychology, sociology, anthropology, and behavioral economy) that have their own methodological approaches but come together to offer immense value to the field. While this could also lead to disagreement and chaos, most professionals she’s come across have been in favor of codifying and bringing a structure to their approaches. Over the years, working with experts from all these disciplines has truly widened her perspective and skillset in the discipline.
When to use which UX Research methodology?
Formative research is used to generate new product ideas, while summative research is used to evaluate a designed product. Formative research is carried out before beginning the product development to understand users’ pain points and figure out what exactly you can solve for. Summative research, on the other hand, is used to test the efficiency and efficacy of your product design, before spending engineering bandwidth building it out entirely.
Another way to look at formative vs summative research – formative research is useful when the researcher’s purpose is to fix users’ problems with their product to make it more user-friendly. Formative testing should be conducted multiple times over a product’s design lifecycle and should be perceived as a directional compass for “what next” in the product from a usability lens. For example, usability testing of the feel and control of a working prototype of an IoT device is a means of formative research.
Summative research is useful when a product’s design has concluded. After completion, product owners typically try to understand what parts of a product are “high-utility” and the ones that aren’t. This research is best done after designing the product but before spending engineering time building it. An example of summative research is when 15-20 users test a mobile handset for features like battery life, gameplay experience, etc. before the device is launched to the public.
Brooke emphasizes that when looking for what to build next, you must carry out formative research, and to assess your product stickiness, summative research is the way to go.
How do new businesses get started with UX Research?
Brooke advises new businesses not to hold back on UX research due to the lack of a large enough user pool. Whether it is your friends or family, you can get anyone who matches your typical user persona to review your product (or its prototype) and share their feedback. What’s important is to pick people who you know will not shy away from telling you the truth.
Brooke stresses the fact that when you are starting out, it is natural to want to hear just positive reviews about something you’ve poured your heart and soul into. However, fighting that urge, as well as the urge to tell people how to use your product, is the hardest part. It is important for you to fight the urge to intervene; let the user figure out your design by themselves and tell you their honest opinion.
Over time, Brooke has learned to embrace negative feedback as she feels it opens so many windows of opportunities for improvement.
The more different we are, the more similar we are
Brooke is a firm believer in the tenet that we have a lot more in common around the globe than we think. She realized that dreams of having a decent job, doing well for one’s family, saving up enough for a nice vacation, being a respected member of the community, are universal no matter which part of the world you go to.
She emphasizes that the hustle to do better for themselves and their loved ones is something she’s found common in people from every country she’s ever researched in. She finds such deeply universal traits uplifting and comforting.
How to make the most of a UX Research program
Brooke highlights that it’s important to start soon, yet go slow and not expect instant or continuous success. While it’s imperative to get feedback as frequently in the design life cycle as possible to confirm if your efforts are going in the right direction, you needn’t test every single product or feature, every single time. Instead, carefully assess the feasibility of your research program and learn to embrace its imperfections.
Must-have tools in UX Researcher’s stack
- The primary tool that a UX researcher cannot do without is a natural sense of curiosity. Brooke feels that the urge to know and understand people and why they perform a certain task in a particular way is the single most important thing for any UX researcher.
- Brooke highlights that the best researchers she knows have a high degree of self-awareness. This means that the clarity that they have about their minds and biases is borderline frightening. They know that they see the world in a certain way, and also acknowledge that maybe everyone else doesn’t.
Building UX for trust and authenticity
Conducting UX Research in a niche market
- Brooke advises that if it is unfeasible to test directly with your target persona, one can break down the roles and responsibilities of the person in question and then look for the next best representative or proxy who also meets the criteria.
- Another approach is Snowball Recruiting, wherein you can ask participants to connect you with their network where you are likely to find similar representatives.
Brooke’s UX reading list
The following are some books that Brooke seeks inspiration from, in her typical day at work:
- Don’t Make Me Think – Steve Krug
A guide on the principles of intuitive navigation and information design.
- Rocket Surgery Made Easy – Steve Krug
A guide on a realistic approach to usability testing that anyone can easily replicate for their site, application, or other product.
- Nudge – Richard Thaler & Cass Sunstein
An easy-to-understand take on the brain’s decision-making under uncertainty and designing environments that leverage how the brain functions.
- Thinking Fast And Slow – Daniel Kahneman
Insights into how humans think, how they make judgments and decisions, and how to unlock the benefits of slow thinking.
Here’s the transcript of the entire webinar. Read along to get a sneak peek into the world of UX research from the maestro.
[00:00 – 01:14] – Introductions
Thank you, Aniruddh. I’m really happy to be here. It’s a pleasure and an honor, and I am so thankful for being invited.
Brooke’s Professional Journey and Role at WhatsApp [01:22 – 07:21]
Aniruddh from VWO:
Without further ado, I would love to learn a bit about yourself and your professional background – how you got into UX research, and how you landed up at WhatsApp?
I feel really lucky. The first thing I’ll say is that I had no master plan. When I turn around and look back at the path that I came, I wonder, how did I get here? I’ve been doing research and design for a little over 20 years. I was in graduate school in New York during the.com boom. It was the tail end of 1999, 2000, when I started working at startups. While I was in graduate school and at the time there was no real differentiation between design work and research. You did everything. So, I got trained to do everything. And over the years, as I changed jobs, sometimes it was because I got laid off (after the .com bust, I got laid off like thousands of other people), I had to figure out – oh God, what do I do now?
So, I just had to go step-by-step. Okay, what’s the most interesting opportunity? Sometimes it was like – What’s the next job I can get? Because I need to pay rent. So, sometimes these were not the most sophisticated, glamorous decisions that I made. But over time, one thing that has sort of rung true is I’m always trying to learn new stuff wherever I’m at. Either I get to work with people who are smarter than I am or have different skill sets. I kind of get them to teach me things. So, over time, I’ve had these opportunities. But it’s not a very exciting story, is it?
Maybe it would make sense for me to talk a little bit about how I got started at Facebook and then eventually landed at WhatsApp. So, I had gotten a call a few years ago from a former colleague of mine who I’d worked with on a project at American Express – a guy I really liked working with. And he said, “Hey, what are you doing? I’m at Facebook, come work with us”.
To be honest, at first, I wasn’t that interested. It’s a big tech company, I’m a little bit older, and I think that’s for younger people. Instagram, that’s for younger people. But then, I started having conversations with them, and every time I had a conversation with them, it was sort of more and more interesting. We spoke about what kind of stuff I would be working on.
They kind of pushed all the right buttons and they recruited me for a team in London that would be focused on emerging markets. As a researcher, this is the most interesting kind of research to get to do – to spend time in India, Indonesia, Brazil, and Mexico, and to really understand what’s going on. So, it was a really amazing opportunity. I was really grateful for that. Then I moved to London and worked on the Ads – Emerging Markets team for about a year and a half.
After that, I kind of wondered what’s next for me. I am one of those people who needs to be challenged and constantly learning something new.
I started to look for other roles within the company. Because within Facebook, you can transfer teams, and there is a tonne of opportunities. There’s Libra, Oculus, Portal, Instagram, WhatsApp, and so many places you can go to. WhatsApp had just decided to build a big, robust research team, and it was the perfect opportunity for me. It’s a big product in countries I am so in love with, interested in, maybe have lived in, or have spent quite a bit of time in. So, the chance to be one of the first researchers to come in and start to build
some of the foundational research was something I couldn’t say no to. I was very lucky to have the opportunity and be at the right place at the right time.
Aniruddh from VWO:
Amazing. For a UX researcher, I think there is no bigger satisfaction than seeing your work reach billions of people, and that is one of the perks of the job and working at that scale.
It is. And it’s a scale that very few of us get the opportunity to work on. I never thought of it as a goal because it’s not so realistic. When I was working on the Facebook side, I was focused on very small advertisers, the people who run the shop that’s around the corner or maybe sell food on a cart. These were very small businesses. So, that was not a huge scale, but they could have a large impact.
But at WhatsApp, we’ve got over 2 Billion monthly users. So, then there’s the pressure of ‘Don’t get it wrong, Brooke, because you’re going to mess it up for lots of people’. But I’m excited that I get to be a part of this.
What Keeps Brooke Motivated and Excited About Her Work [07:22 – 09:41]
Aniruddh from VWO:
I have a follow up to that question. You have been in the field of UX for almost the entirety of your career. So, what is it about UX that keeps you interested and coming back to the job every day?
I feel like I’ve got the best job in the world because I’m someone who’s a little bit nosy and someone who definitely likes to learn new stuff. I’ll get bored if I’m just doing the same thing all the time. And it turned out that this is a job that really meets those needs. I get to learn new stuff all the time. Earlier in my career, it was about building my skill set. Oh, can I do usability testing? Can I learn how to do a card sort properly? Am I recruiting the right people? Let me get good at that. Let me get good at doing in-depth interviews and all these things.
And then, as I’ve gotten a little bit older, now it’s about, can I think more broadly? Can I have more vision with the kind of work that I’m doing? And so I get to be tested and learned. It’s really a little bit selfish, right? That I get this sort of personal satisfaction.
The other piece of it is that I have this amazing job that on paper is very sexy. I’m one of the few researchers who works at a company that will pay for me to get on a plane and fly for hours to go spend time and talk to people. Sometimes I even get to bring teammates with me who aren’t researchers.
So, the opportunity to bring an engineer with me who is writing code for the product that we’re working on, for her to have a chance to come to the field and talk to people who are using a product that she built, it’s very rare for her to have that experience. And this really could be life-changing for engineers, product managers, or data scientists who I bring in the field with me, and they can really connect and see that the work we do really does impact people. So, I love being able to be a part of that and get to see it. It’s a little bit like a holiday. It’s like Diwali or Christmas. It’s like someone’s opening a present, and they’re so excited.
How Brooke Approaches UX When Building For Such a Diverse Audience [09:42 – 15:00]
Aniruddh from VWO:
Amazing. So, in this pivotal role at WhatsApp, how do you approach UX when you’re building for everybody across so many personas and so many jobs that people do with WhatsApp. WhatsApp is a classic use case because right from small businesses to friends, to people in a relationship, I think a lot of people use it. So, how do you approach UX when you guys, when you’re building for such a diverse use case?
Yeah, it is this very important thing that we take incredibly seriously. So, we as a company and as an ethos are not trying to move the fastest to get the newest thing out. WhatsApp in particular, we will maybe even go a little bit slower to try to really think through things before we release them.
From a research perspective, I have to think about, how do I support that? How do I support the team to make the right decision? So, that means with an app that has 2 Billion monthly active users, I can’t just go like, ‘Well, you know, I’ll just go to one country for a week and I’ll talk to 20 people, and then we’ll know the answer, right?‘ Because there’s such diversity in our user population.
We’ve tried to build one product for every run. So, I am trying to make sure that we’re talking to users all over the world, particularly in our key markets, in the key countries that use WhatsApp the most.
Aniruddh from VWO:
So let’s say if you come across different needs or different kinds of users. Then you have to make a choice, right? You cannot put everything there. So then how do you prioritize? How do you use the real estate of the screen? What features take priority over others? How would you decide that?
A lot of people get involved in those discussions. And those discussions can take place over weeks and months and even longer. I may be the researcher generating the research, but we have product marketing that is looking at markets and opportunities. So, they come in with their own research and data. The product management team absolutely drives a lot of this discussion from their experience. Product marketing may come actually in the field with me. The engineers have a seat at the table, and so does data science. So, we are kind of all together, what we call cross-functional teams, having these discussions from different points of view and sort of weighing through the pros and cons of things and then prioritizing.
What gets prioritized, sometimes the criteria for that can change, but it’s really based on where the biggest need is, where we can have the most positive impact? Is it technically feasible? Are there government entities we need to work with to get permission to do things? So, it can be quite complex. And there’s a lot of people at the table. We just kind of go through and go through to kind of winnow down. What are the right things? And then, something like a pandemic happens, and the world completely shifts. So, then as a business, we have to react and say, “Okay, what everyone’s needs were six months ago, might be quite different today. How do we then change gears as quickly as we can and try to focus on that?” Because there’s just some stuff you can’t, you can’t plan for.
Aniruddh from VWO:
Yeah. Awesome. And I think that’s the hallmark of a great product where different teams come together, and that’s where a lot of great ideas get produced, just from the seeds of initial ideas. Great.
Sometimes people will come to me as the researcher and be like, “Well, you know all the answers”
So, it absolutely is this very collaborative kind of thing. And if one voice in that is missing, it makes everything a little bit less. And so it’s great to have everybody’s point of view because we can tackle problems from a lot of different angles.
How UX Research Has Evolved Over the Last Two Decades [15:01 – 19:53]
Aniruddh from VWO:
Great. Amazing. So, on that point itself. How have you seen the UX methodologies evolve over the years and its adoption across the industries?
There’s been a good amount of change, I think, in a very positive direction. When I started out, UX wasn’t even a phrase. Your title was then ‘Information Architect’, and usability testing was one part of what you did. There might be some other methodologies in there, but the first thing you ever learned was how to run a usability study, right? So you’ve got a prototype, or maybe you’ve got designs on paper and you need to put them in front of people and ask them to do things and then see where the design works and where the design doesn’t work.
I’ve learned to do a few more things than that from that time. And so over time, what’s been great is that the umbrella of UX really embraces so many different backgrounds and industries.
Over the years, I’ve gotten to work with more and more psychologists, sociologists, anthropologists, and behavioral economists. All these different disciplines come together, and they bring their methodological approaches and so there’s always someone who knows how to do this one thing really well.
I worked with a woman years ago who wrote the best recruitment screeners I’d ever seen. And so I was like, ‘I’m going to learn everything I can from her’. She just always got the best recruits. She got the best participants. I always thought she’s doing it right. And over the years, as I get exposed to these people who have these different backgrounds, my toolbox expands. So, from a selfish perspective, I’ve just gotten to learn a whole lot more. I’ve seen the discipline grow, and I was trying to really codify. Having all those different disciplines there can create chaos and disagreement about things, but I’m seeing more and more folks wanting to codify and bring more best practices and structure to things. And so I think that’s mostly a good thing.
Aniruddh from VWO:
Okay. Can you share your personal experience of how you’ve seen companies adopt the practice of UX research? Because I think a lot of companies never put that emphasis on it. And I think now more and more companies are putting users at the center of their entire product experience. So how have you seen that evolve?
Well, I could just tell you a couple of little stories from early in my career to now. So early in my career, I can remember being in meetings with other teammates asking me what I do, and kind of being very dismissive. I was working on a project in 2000 for a big bank in the United States, and we were trying to build a large payment engine that would allow multinational and cross-national sales and delivery of products and goods on a very large scale. So, if you want to buy a hundred tons of lumber from Brazil, you need it shipped by a Dutch shipping company, and it’s going to be delivered to Mumbai. There’s a lot of different companies going on.
We were working on a payment engine for all of that. I remember being in a meeting with a guy who was managing the engineering team, and we were talking about the features that the product needed to have, and one of them was search. He thought the search was going to be too hard for them to build. I was shocked. I remember saying that we’re going to have people who have invoices and order numbers, invoice numbers, and payment slips. They will all have these different unique numbers on them, and there’s going to be hundreds of thousands of them, and they’re going to need to be able to find their invoice to pay or request payment on and all these things. And he said, ‘No, it’s just too hard, it could just be in a dropdown’.
And, we didn’t quite yell at each other, but it was this allergy that he had that it was too hard to build, so that’s too bad. My perspective has always been that how are people going to use this thing, they’re not going to use it if it’s so painful.
I have really seen a shift over the years, though. One of my favorite people to work with at WhatsApp is an engineer. Part of what I love about working with this guy is that he’s so into research, he wants to build a good product and is always asking me what more I can tell him to help him make the product better. It’s this incredible embrace of the value that research can bring to an organization. So, that’s a huge swing from someone saying, ‘Nah, it’s too hard’ to ‘Tell me more’. It’s lovely. Not all companies are like that, but that’s sort of been the progression for me.
Types of UX Research and When to Use Which [19:54 – 22:06]
Aniruddh from VWO:
Great. I’m going to ask you a little technical question. How do you see formative UX research, how is it different from the summative approach and which one would you recommend?
Formative research versus summative research is not one or the other kind of answer. So, formative research, let me just give a little bit of background definition for everybody.
You want to make sure you build the right thing, right? So, there would be types of research that have many methodologies underneath them that you’d want to actually use on a full design life cycle.
If you’re trying to figure out as a team, what to build next, you want to do formative kind of research where you’re talking to people about what kind of problems do they have when they are experiencing X, Y, or Z because that’s going to give you ideas around what are the problems you could solve for this. But the summative stuff – usability testing, any kind of product-market testing, click test time on tasks, see how efficient and effective a product is, that’s going to be types of summative research. So, I think you have to do them all.
Aniruddh from VWO:
Okay. So let’s say when somebody is starting out, you would recommend taking the formative approach, and once you get deeper into the UX cycle, then you’d rather use the summative approach.
Yeah, because if you’ve got a prototype or designs in Photoshop that you want to get feedback on, you probably don’t want to generate too many new ideas at this point. You want to just evaluate the stuff you’ve already done – Is it right? Is it not right? What needs to be fixed? Where does it break? That kind of stuff.
How New Businesses Can Get Started With UX Research [22:07 – 27:26]
Aniruddh from VWO:
Also, a lot of our viewers who might be listening to us might be working for businesses that are still getting started, that are still new, or they might be launching new products if they are in established companies. So, my next question is about them and for them. So, user research is kind of very hard to get right for new businesses, right? Are there any frameworks that you use or one you’ve developed yourself to find answers to research questions that you ask participants? As in, how can new businesses get that research part right?
Yeah, it’s hard. Especially, if you’re just starting out, and my first answer is that something’s better than nothing. You do your best. So, if you are building something at night, on the weekends and you think it’s going to work for people, try to find people in your friends and family community who represent your typical user. And then, put the designs or the prototype in front of them and then ask them, ‘I want you to try to buy a pair of shoes on this’, that is if you’ve made a small e-commerce app or something. It’s not going to be the most rigorous thing. That’s okay. But, you just want to get people’s input on things.
I can tell you the first few times I tested stuff that I designed, and people didn’t like it, it was soul-crushing. And I may have cried but maybe not. But, that’s part of the process. You can build a capacity to handle that kind of criticism because the whole goal is to get some people first to test your products, so they help you find all the mistakes before you release it. So, instead of having everyone hate it, it’s good to just have a few people.
Aniruddh from VWO:
And, this is, I think what you just mentioned in response to the question before that, is the summative approach, right? What if somebody is just getting started, they haven’t yet built a prototype and it’s just an idea. So, how would you get started then?
Talk to people who you think might be potential users of the product and don’t try to sell them on your idea, talk to them about where they have difficulty in the process of whatever they’re trying to do. For example, if you learn from a parent that when their kids come home from school every day, their shoes are filthy, and cleaning them is pain. So, maybe you’re trying to build a product to make shoes don’t get dirty so easily or it’s easier to clean them or something like that. So, you want to talk to people who have really dirty shoes all the time, right? What are the types of jobs that have dirty shoes?
Go talk to people and ask them about their biggest problem. And listen very attentively. People will tell you their story and will open up like a flower to you if you really are actively listening to what they say, respond to the things they say, and follow up with the things they say. So, treat these people as precious because they really are, and pay attention to them.
Also, you want people to complain to you. So, you want to try to make them comfortable enough so they’ll do that. Because pain points, problems, complaints, that’s very fertile ground for product development. For the development of ideas and to solve problems. Because you have to know what the problem is first. So, talk to people around who have the same problem or a similar problem to what you’re thinking about.
What Years of Research Have Taught Brooke About Universal Human Behavioral Traits [27:27 – 31:26]
Aniruddh from VWO:
Great. And after so many years of understanding how humans make decisions, what are the behavioral traits you have identified, which are common across people?
When I’m out in the field or when I’m talking with other researchers, I usually refer to the fact that people have a lot of hustle too. They’re just trying to do better for themselves, do better for their kids. And I see that over and over and over again without exclusion, in every country I’ve ever done research in. So, for me, that’s very uplifting and comforting. It creates a lot of love inside of me for the world because I see just how much people are trying to do their best and make a little bit better life than they had and for their kids. So, I think those things are deeply, deeply universal.
Aniruddh from VWO:
So, I think empathy as a skill is very, very handy in certain situations when you understand the actual pain of the end-user and then try to emulate that in whatever product or service that you’re trying to build.
Yeah, a hundred percent. Empathy is such a buzzword right now, but it just means – ‘Can you feel what it’s like for somebody else?’ I was born and raised mostly in the United States. English is my first language. I’m very well educated, and there’s a lot of privilege in my life. So, okay, I have to understand that about myself. And to be a good researcher, understand the biases and cultural baggage that I have, right? You’ve met Americans before. You can tell they are American, of course.
When I’m coming into another country where I’m not from, I have to really think about how to put myself in the shoes of a 19-year-old, married woman who’s job is taking in laundry for the neighborhood and she’s got a young child she needs to make sure is healthy and gets a good education. She and I have very different life experiences, but it’s really my job to understand what they feel and care about.
I call this my 90-minute love affair, or however long my sessions are with participants, where I’m not trying to argue with them about what they do and how they have their life. I’m just really trying to feel what they feel and care about what they tell me is important to them and ask them why it is important to them and then just really listen. That helps me build that empathy. It’s something that everyone can build. We certainly all have different starting points. It’s easier for some of us than others, but everybody’s got it.
How to Make the Most of a UX Research Program [31:27 – 33:45]
Aniruddh from VWO:
Great. So, from your professional experience, how would you suggest that businesses design their research programs to get the best value out of it?
Yeah. It’s so hard. I would just say the first thing is to just start somewhere and know that it’s going to grow over time.
There will be times when as a business, you have to make a decision where you know that this is best for users, but because you need to monetize it, you’re going to have to do it the other way. At least make those decisions consciously, and by being fully informed, I would say.
Don’t expect to do all the kinds of research on every single product every single time. It’s not feasible, start slow. I would say the best place to start is that if you’ve already got something designed, or you’ve got some idea in your head, sketch it down on a piece of paper and start showing it to people and then ask them to search for something on it or show you where they would tap. On the piece of paper, you could just see whether they tapped where you put the search.
If they did, that’s good. That means that you probably put it in the right place, and it looks like it’s meant to look. If they’re putting their finger on lots of different places, it doesn’t mean that they’re wrong. You’ve got to understand that. You’ve asked them where do they think it should go? You have to take that into consideration that however you’ve shown the search, it’s not making sense to them. So you need to go back to the drawing board. If you can come at it with that kind of attitude, you’re going to be miles ahead of lots of other companies.
Must-Have Tools in a UX Researcher’s Stack [33:46 – 37:48]
Aniruddh from VWO:
Great. Thanks a lot for that insight. I would love to also talk to our audience here. 230 plus people are listening to us right now. So guys, if you have any questions, do keep it in comments, and we’ll take as many as we can probably be at the end of the session. So, before we go ahead, I think a lot of people who are listening to us might be starting out in their career, or they might be just growing and budding, and they’re still learning a lot. What would you say would be the most important, must-have tools in a UX research toolkit. Without naming any specific names, but just in general, what tools should one have or use in order to do good UX research?
I’m lucky that I have had the fortune of living in South America. I’ve lived in Asia. I’ve lived in Europe, and so I have been able to live in cultures that are not what I grew up in. When you live in a culture that is not the one you grew up in, you start to learn a lot about stuff like, ‘Oh well, that thing we did when I was a kid, I thought that was normal, but they don’t do it like that here’. You can start to identify where you have biases, whether they’re gendered, racial, or cultural.
The best researchers I know, know themselves and their biases really well. I’ll give you a funny little example. So, in the United States, avocados, the fruit, some in America actually call it a vegetable, but it’s a fruit. We all know this. In the United States, avocado is used for savory dishes. We put it in salads, we make it into a dip. We can put chips into it, but it’s all used for something savory. You put it with eggs, that kind of stuff.
Fast forward to me spending some time in Indonesia, and I love avocados. So, I learned one of the most delightful things, Indonesian culture and lots of Asian cultures actually use avocado as a sweet. And I thought this is the greatest thing ever. I can have avocados for dinner and dessert. Eat it with condensed milk or turn it into a Popsicle, a sweet Popsicle. I didn’t even realize, I had this bias that avocados were only for savory until I got exposed to the fact that they don’t really have to be, right? And as you kind of learn those things, it’s really helpful.
So, the best researchers I know, one of the biggest pieces of their toolkit is knowing themselves and knowing that ‘I’ve got these ways that I see the world, but maybe everybody else doesn’t. And can I be loose about that? Especially when I’m doing my research?’
Aniruddh from VWO:
That was a lovely story. One lesson for me was to keep our biases outside the research room. And because I think users can delight you in more ways than you can think of and they would come up with use cases that you can never even imagine.
Attendees’ Questions Answered [37:49 – 58:03]
Aniruddh from VWO:
Amazing. So let me get a couple of questions from our viewers here. So, the first question is from Autumn. She says, ‘I do CRO, but want to take up a junior UX designer role. Every job I go to wants a graphic designer who knows UX principles but then can code and build tests. What’s the best advice moving forward?’
I’m probably not going to have very good advice because I share frustration, Autumn, with this idea that you have to have all these incredible skill sets. I’m one of those people who think that design, research, engineering are very different skill sets. So, when a job description asks for what we call a unicorn, and I don’t use that as a compliment, I get kind of upset about it, it’s unrealistic. It doesn’t give people the opportunity to really grow into a specific skill set. That said, there is sometimes a lot of opportunity at smaller companies that need someone who’s a Jack of all trades. So, if you can learn a little bit to kind of be dangerous, over time, you’ll be able to grow into things.
So, I think smaller companies that need people to wear one or two or three hats could be a good experience, but it really depends on what they’re pushing you at. I would say, don’t try to learn something that you don’t actually care about because it will show in your work. Don’t kill yourself for that. But, this is a very American point of view. And I’m also way deep into my career, and I sometimes forget what it was like when I first started out. When I first started out, I took what I could get. And, if I think back while I was in graduate school, the first way I got into working at a startup was I took an unpaid internship.
I just showed up for free to do some work so I could get some experience. It was hard. I didn’t have any money, I had to try to work a job on the weekends, and it sucked. But, those things built over time. So, that got me in the door to work at a paid internship, while I was in graduate school. I was like, ‘Oh my God, they’re giving me money to come here. This is the greatest thing ever’. So, my expectations were a lot lower earlier in my career because I knew I just needed experience. I think if you’re a junior person, be the best junior person you can be. You’ll be a senior person one day. Don’t worry about that, you know? But if you can get a place that you feel is going to be good and let you try things, that might not be a bad idea. I don’t know if I answered your question, Autumn.
Aniruddh from VWO:
So, the lesson here is to take what you can get and build on top of it.
I mean, the earlier you are in your career, the more it’s about just getting your foot in the door. As you mature through your career and you start to decide what you really like, what you don’t like, what you are good at, what you are not good at, where you want to focus, then be more selective. I’m not saying, just be thankful. The job should appreciate you and all the effort you put into it, and you should feel that.
I think I see a fair amount of folks in the States who have no industry but are very particular about what they want to work on and not work on. And when I’m sitting on the other side interviewing them, that doesn’t come across well because, with no experience, they want to be the one who calls the shots. When you don’t have the experience and maturity to make wise choices, you’ve got to learn.
Aniruddh from VWO:
The next question is by Kartikey. ‘How can we, as a small business, understand our audience when we don’t have enough data?’
When I use the term ‘data’, that can come in both quantitative and qualitative ways, right? And small sample sizes from a qualitative perspective, that’s not a problem. It’s the type of data that you have. So, if you’re talking about just behavioral metrics on a platform, that’s quantitative data, right? And it can tell you what’s happening on the platform and you can see some things. You can see general trends. Where people are dropping off if you’re selling something, where they are dropping off in the funnel.
If you can see that people are getting partway through clicking through products and then they put something in their cart and then nobody converts, or very few convert, that data tells you there’s something going on in that cart experience. So, pay attention there. It may not tell you what.
You have to use qualitative research to get in at that. So, get 10 users, 20 users and bring them in and have them talk to you for 30 minutes or an hour, have them take something through the cart, be quiet, let them try to buy something, and see what happens to find out why they aren’t converting. So, you don’t have to talk to like the volume that I’m talking to.
Aniruddh from VWO:
The next question is by Suruchi, and I think that’s one question that a lot of people might have. ‘How do you build for trust and authentic information on WhatsApp? It can be extended to any of the platforms.’
Yeah, well, trust is this incredibly complicated topic and I have had a keen interest in it for my whole time at Facebook and WhatsApp. I actually used to chair a monthly call of researchers or over a hundred researchers who were focused on different aspects of trust. So, I’ve done research on trust. I’ve worked with a lot of other researchers on trust. I’ve talked with industry experts about it.
The thing I know about trust is that it is so hard to gain and very easy to lose. So, one of the things that I learned from another researcher that’s really resonated with me and I think really makes a lot of sense is thinking about trust in terms of making it easier for your users to make sense of things that they feel are untrustworthy. I’m going to get a little nerdy with how I talk about this.
I would flip it around and anchor it in the negative. It’s much easier to focus on what are the things that cause distrust, and how do we eliminate those, right? What are the things that cause distrust in the technology or the other people on the platform or the content that’s on the platform? What causes distrust or mistrust and focus on those because those are solvable problems.
When we talk about trust signals on a platform, they can be different from person to person. But usually what people are looking for is whether it’s a real person they’re talking to on the other side that has my best interest at heart. Are they authentic? Are they going to do what they say they’re going to do? And if they don’t, is there some recourse where I will have some power in that? But it can really depend on the platform and what you’re trying to do to figure out what trust signals are. So, it’s really simpler, and you can have more actionable things to do if you focus on making sure things that are untrustworthy, you could try to eliminate those.
Aniruddh from VWO:
Sure. The next question is about something you must have faced as well, probably earlier in your career, not as much now. The question is from Shirley Patrick. ‘How can you make marketing include UX into the discussions? And by include, she really means include, because typically there’s somebody from UX who might be present but their voice may not be heard as well. So how do you ensure that you include that person into the discussion?’
Shirley, that’s a great question. It could even be expanded to, how do we make sure that UX or research gets to be a part of any of the conversation around product stuff? I’m going to probably give a boring answer.
The places I’ve been where I’ve been able to have the most impact, one of the things it really comes down to is interpersonal relationships, the hard stuff that takes a long time. Part of the ways I’ve tried to help is that if the marketing team is like, ‘Yeah, yeah, Brooke, go away’, then
maybe part of what I can do is figure out what the marketing team is up against or maybe I represent something that is scary to them. So, I try to understand why they have the approach that they’re having that may not be including me because I want to be included.
So, what can I understand about where they’re coming from, right? Also, can I demonstrate value? Can I solve some problems for them? Because there’s a bunch of stuff that market research can do, right? Such as opportunity sizing, profit product, market fit, but there is stuff that UX research can bring to the table that others can’t. What are all the nuances of it? If we do this, then what kind of reaction will people have? So, can I bring something that’s valuable to them, to the table and trying to sort of slice that in.
At some companies, you need to find a champion who’s more senior, who you can kind of, get to be an ally to get involved. Maybe senior people have to talk to each other to say, ‘Hey, everybody needs to play nice together’.
It can really run the gamut. And I think a lot of it’s predicated on the culture of the organization that you’re in. Because some have just got these barriers and walls that teams don’t work together at all. Marketing does marketing. Engineering does engineering. We just throw stuff over the wall. And if that’s the case, it’s a lot harder. It might be a more hierarchical organization, and so you might need to kind of go up the chain to be able to go over and back down the chain.
Aniruddh from VWO:
Okay. The next question is from Nahush. The question is, ‘How do you do user research with enough confidence in a niche market? And it’s not really a less number of users, but a niche market where it might be difficult to find users to talk to and test with?’
This is a great question, and I’m glad it got asked because it’s happened to me before too. So, at WhatsApp, we’ve got a big user base, but I’ve worked on products that are very niche. Maybe the user of the product is a chief financial officer at a company, and they’re notoriously difficult to recruit because they make a lot of money so you can’t offer them a bunch of money or they might not be allowed to be paid because of whatever kind of laws are going on. So, how do you find this? You know, beg, borrow, and steal.
But then you can also think about proxy or representative users, right? So, if I can’t get my hands on a CFO, and I’m just making that up as the niche, person or group you need to get at, then is there someone maybe a little bit down the chain, but he’s still pretty senior that I could get at it widens my circle.
Essentially it’s a recruiting pool that we think about. So, what roles would have like behaviors? If you’re looking for someone who’s a decision-maker for X, right? Well, so break it down. If they’re a decision-maker for X or they are in charge of, you know, a Z amount of budget, maybe you can start to look at breaking those things down, and it might widen your pool, so you get more representative. Not exact. That’s really the way we do it.
There’s also another way – if you’ve ever heard of snowball recruiting. If you can find one person who represents the type of people you need. This, I think, comes from anthropology. You will ask the participant, ‘We’d like to speak with other people, could you tell me three or four other people who you know, who do this kind of thing and ask if we can contact them please?’ So it’s like a snowball, as it rolls, it gets bigger and bigger. That’s why it’s called snowball recruiting. So these are some ways to do it.
Aniruddh from VWO:
Okay, great. The next question is from Kuda. The question is, ‘Do you also do quantitative research, like AB testing for new features?’
Yeah, so I am considered in the Facebook/WhatsApp world, a qualitative researcher, and we have kind of our own definitions of that. I do what I call baby quant. So, l do A/B testing time-on-task testing, first-click testing. One thing I did last year in a bunch of African countries is we had an advertisement type, and we had to figure out what the right label needed to be on a call to action button. We had a couple of different ideas, but we were not sure which one was going to get more clicks and increase more revenue. So, we A/B tested that and I put a few qualitative type questions in with that research.
So, we figured out which one was better based on what people clicked to do X, Y, Z. But then, I would also ask them which one they would prefer out of the two and why? Because that ‘why’ is really, really useful because we have to make other design decisions after we’ve done this research. That’s like golden information for us to pull from, to make wise design decisions afterward or development decisions.
Aniruddh from VWO:
Yeah, we have a lot more questions, but in the interest of time, I think we can take two or three more. The next question is from Ankur Sharma. The question is, what books are you currently reading, and what are some of the favorite books around UX that you would recommend?
I read a lot of fiction for fun. Right now, I’m reading a bunch of science fiction stuff, which is not everybody’s thing. I will say when it comes to UX, a few things I would really recommend – Steve Krug’s, ‘Don’t Make Me Think’. It’s a really easy book to read if you’re getting started and you want to figure out something like – ‘How do I run a usability test?’ I just think he has a lovely way of breaking things down and he’s also a really nice person.
At a conference one time, I gave a presentation, this was years ago, I was super nervous and he was in the audience. I saw him, he came up to me afterward and he spoke to me, and he gave me a huge compliment. He’s just a lovely person too and it’s very encouraging for people getting started in their career.
So, I think very highly of him. And the way he breaks things down, and he makes it much less complicated than it needs to be. So, ‘Don’t Make Me Think’ or ‘Rocket Surgery Made Easy’. His two books are good.
On the other end of the spectrum, if you are looking for really nerdy stuff, for the last few years have been quite interested in behavioral economics, and so I would say pretty much anything that Richard Thaler or Cass Sunstein has written. Richard Taylor wrote ‘Nudge’, and Cass Sunstein has been a co-author with him on a bunch of stuff. The way he writes is really accessible, and you can understand he’s talking about decision making under uncertainty. How does the brain make decisions so you can design things that take advantage of how people’s brains actually work, and they want to make decisions?
If you want to go a little bit deeper, then go to Daniel Kahneman. He wrote ‘Thinking Fast And Slow’, but don’t start there because it’s a hard book to read. Those are just off the cuff suggestions.
Also, there’s a great woman, and she happens to be Indian American, Sheena Iyengar. She is a research scientist at Columbia University, who has written a bunch of books about choice and decision making, and she has one of the sharpest minds when it comes to understanding. She’s a psychologist by training. She’s got a couple of amazing Ted talks. I’ve actually seen her talk live. She’s so sharp and just such a crisp mind, I really love her. So, I want to recommend anything she’s done.
Aniruddh from VWO:
The next question is from Magdalena. The question is, ‘How do you measure your product design and what are the best practices that you would recommend?’
Well, so we look at a few different things because I think essentially what you’re asking is, is it good design, is it the right design? And so as we’re going through the design life cycle and we’re doing maybe usability testing on it, we’re looking at factors like ease of use, satisfaction, do people use it and walk away going ‘Yeah, I got done, but I hated it’ or ‘No, that was easy, I like that’. Or you know, describing it as simple to use or not. And not all products have to be simple to use. You can build really complicated software that has to be complicated for a reason like flying a helicopter. So, there may be time on task, how effective people are, can they actually get the task done?
And then measures of efficiency – how long it takes them to get things done. Satisfaction – how good or not good did they feel about the process they had to go through to finish the task? So there are all those kinds of things, and you can keep doing those. Once we release a product though into the market, we’re looking at stuff like, if it’s an app, how many people are downloading it? How many people are using it daily, weekly, monthly, how many complaints are we getting, and what is the nature of the complaints we’re getting of things to help us understand that we just released something and then all of a sudden we got more troubled tickets that come up. And it’s the same theme, and we’re hearing lots of people don’t like this.
So, we got to go back and figure out what went on. It’s really going to be somewhat product specific because we’re going to tie these to business outcomes, right? The business may say these are our goals, revenue goals, active user goals, satisfaction goals. So, we’re going to be measuring against all of those and monitoring and then, looking at them within a threshold to make sure that we were managing all of that stuff.
Aniruddh from VWO:
Amazing. I think there are about 40, 50 more questions we’ve received. Since we don’t have time to cover all of them in the session, is there any way that these people can reach out to you in case they have questions?
They can, I will just say be gentle with me. I get a lot of requests on LinkedIn. I will do my best to get to you as soon as I possibly can. But, if you want to reach out to me, make sure you have a crisp, sharp question, and I will do my best. You have to imagine, with COVID 19, WhatsApp is a little bit busy right now. So, my day job is keeping me a little bit busy. You can also just follow me on LinkedIn or connect with me there. That would be great. Happy to.
Aniruddh from VWO:
Great. Thanks a lot, Brooke and thank you all the 200 plus people who joined us for the session. I’m sure they got to learn a lot from you, and I got to learn a lot from you. So, thanks a lot for that and guys, tune in for the next session of Masters of Conversion. Check your email boxes. You will get a notification very soon and thanks a lot for watching. Thank you.
Thank you. Bye, everybody. Thanks, Aniruddh. I really appreciate it.
While UX research in today’s age is no blackhole, it needs a clear distinction to separate the noise from the signal, and voices from industry-leaders such as Brooke keep the domain evolving at the breakneck speed it does. Whether you’re just starting out in the field of UX, or are a seasoned professional, Brooke’s timeless advice is something to internalize and live by. We hope you found this edition of Masters of Conversion insightful.