Editor’s Note: This transcript was created using AI transcription and formatting tools. While we’ve reviewed it for accuracy, some errors may remain. If anything seems unclear, we recommend referring back to the episode above.
Episode Trailer
“I do feel like more and more companies are starting to want to do experimentation. Not all of them obviously have the capacity or the user base to do it, but certainly that mindset is starting to shift.”
“It’s not really something that is taught—there’s not a course you can take. Yes, you can understand the data points you need, but without having the time and the experience and the learnings from the experiments themselves, it just doesn’t all come together.”
“Don’t put too much pressure on yourself. Everyone’s making mistakes when they start out, and actually that’s exactly what experimentation is about—it’s about failing fast and learning from it.”
“A lot of people get bogged down in the debates—Bayesian versus frequentist is a big one, or should it be CRO, should it be experimentation, should we call it A/B testing or should we call it growth. Yes, it’s important, but at the same time we’re ignoring a lot of the other important conversations.”
About VWO
Welcome to another episode of the Women in CRO series by VWO podcast. This series is an ode to the contribution of women in the CRO industry.
Before we speak to our special guest for this episode, here’s a quick summary of who we are and what we do: VWO is a leading experience optimization platform that helps fast-growing brands optimize their digital experiences. Using our latest product, VWO Insights, you can understand user journeys and identify conversion roadblocks on your website and mobile apps.
So without any further delay, let’s jump right into the conversation.
Guest Introduction
Akansha: Hello everyone, my name is Akanksha and I’m currently working as an account executive here at VWO. Before we dive into the podcast, I just want to take a moment and express my gratitude to each and everyone who’s currently a part of this podcast, whether you’re a new listener or are coming back to us again. I would just like to thank you for your time and choosing to spend your time with us.
In today’s episode, we have the thrill to have Steph. She is a seasoned consultant at Conversion. She has 8 years of experience in the CRO arena and she brings a wealth of knowledge and expertise to the table. As a CXL certified optimizer, she has honed her skills and made significant contributions to the field. Previously, Steph held roles as CRO and UX Manager at Specsavers, further solidifying her reputation as a leader in the industry.
Beyond her professional pursuits, Steph has a passion for staying active. Whether she’s hitting the gym for some strongman-style weightlifting or engaging in Hyrox training, Steph is always striving to push her limits and achieve her goals.
So join us today as we delve into Steph’s journey, exploring her insights, experiences, and the lessons that she’s learned along the way.
Main Interview
Personal Background and Fitness Journey
Akansha: Welcome to the podcast, Steph. How are you?
Steph: I’m good, thanks. Yeah, doing really well.
Akansha: Steph, I recently found out that you like to stay fit and consistent, and that is something that I’ve been trying to do for such a long time. I think all my New Year’s resolutions are around the fact that I want to start this journey of fitness and do something. Yet I think irrespective of how motivated I am, there’s a factor of discipline and consistency that matters more than motivation. So my question to you is—and I’m being a little bit selfish here—how do you remain consistent? What motivates you to be disciplined?
Steph: That’s a good question. A lot of it for me comes down to optimizing my health, which comes with the optimizing nature, but I really like to understand how my body responds to different levels of training and also see how far you can push it. Because there are some people out there that are capable of crazy things—running seven marathons in seven days. So when you look at things like that, it kind of makes you think, “Well, what could I be capable of?”
I guess it comes from that place of optimizing my health. In terms of discipline, I think it also helps my mental health as well, so it’s just there to keep everything in check. It gives you a moment of peace throughout the day. Also, training with friends is really beneficial as well because you get to catch up with them, have conversations whilst you’re working out, and you kind of forget that you’re doing that hard work.
Akansha: How did you get started? Was this something that you were always passionate about growing up, or is this something that you took up recently?
Steph: I started in lockdown. We came back from traveling and I’d put on some traveling weight, so I decided I wanted to lose some and I did Joe Wicks’s 90-day body plan, which was really good—really motivational. Then I kind of just got this urge to keep doing different challenges. So then I ran a half marathon, and then I did a certain amount of miles in a month, and then I joined a new gym and they were doing strongman training.
I felt like weightlifting was something I liked the idea of, but I wasn’t really brave enough to try—the gym’s really overwhelming. So I joined that class and then it just blossomed from there. Six months later I did my first competition, then I did another one, and now we’re doing Hyrox. After that we’re just going to move into a hybrid training style. So it just kind of accelerated year after year, and every year there’s a new fitness target.
Evolution of Experimentation Approach
Akansha: How has your approach to starting an experiment changed over the years? Are there any myths that you previously used to follow by heart that you don’t follow anymore?
Steph: I think the biggest change has been the rigor behind the experimentation and where the concepts come from. If I cast my mind back to when I first started, I was completely new, didn’t really have a lot of guidance, and was just trying to learn on the go. So it was all ideas from competitors or best practice things, and you’d just kind of come up with the idea, write the hypothesis.
Whereas now, fast forward all these years, the rigor behind the way that we come up with experiments at Conversion is so different. Everything is research-based, and not only that, but we have the lever framework and then we also have to think about all of the audiences and the areas and MDE. Looking back, I didn’t know what MDE was at Specsavers—I had no idea. I was just lucky enough that we had big traffic so it was all working out fine.
But it was a big learning curve, and I think a lot has changed in terms of backing up the experiments and also then deciding what the actual execution should be as well. Thinking about: does it fit with the MDE? Could it be better? Why would it lose? All of those things and putting more rigor behind the actual thing that you’re testing as well as the hypothesis.
The Lever Framework
Akansha: You mentioned the lever framework. How do you apply it when conducting these A/B tests?
Steph: It kind of feeds into almost everything that we do. When we first start a program, what we’ll do is an explorative study. We’ll take a number of research methods—that would include user testing, competitive analysis, a survey, possibly more as well depending on the industry—and we’ll write down essentially everything we observe from that study.
Each of those observations can come together to group into findings and insights. Those insights are tagged at a lever level, so everything we then know looks at a lever level. We can say, “Right, well actually we have seven observations about direction, three about delivery, none about process flow,” and so we can start to look at the beginning of the program where we should focus our efforts. We can start to cast a wide net, but only look at things that we’ve validated already with research.
That immediately starts to get you that ROI and also those insights and those learnings to then exploit those levers moving forward. As we gather more information and more research, we’ll fold levers away so we know where not to focus. It really feeds into that whole explore-exploit-fold methodology.
In terms of past that one view of the experiment program, we have this lever base across every single client we’ve ever worked with. What that means is we have 16 years of experiments that are tagged in the same way. So the power of that meta-analysis is huge because it means we can look at what levers generally are working well—say for fashion in e-commerce—and over the years we see social proof kind of taking a bit of a decline. We can start to see those industry trends that can also inform our strategy overall for that client and for those users as well.
Running 100+ Experiments
Akansha: I recently came across your LinkedIn post where you mentioned that you ran about 100+ experiments in the year 2023. That’s quite a lot. How did you do it, and did you observe any particular pattern when you were running these tests?
Steph: When I saw that I was like, “Wow, that’s crazy that I worked on that many in one year,” and I’m sure this year it would be more as well.
I guess how we do it is I’m really lucky that I have a really supportive team. We work in a pod structure—there’s an associate consultant in my team who’s excellent and she works really well, and then we have QA and developers and designers and we have a project manager. So together we form a cohesive team that is well-oiled and works in the best way and is very streamlined.
In terms of trends, I mentioned social proof. We’ve seen the results of adding reviews or talking about Trustpilot slowly decline over the years—the average uplift that we’re seeing. I think a lot of it comes down to the savviness of the consumer now understanding a bit more about how they can be manipulated online and not really trusting necessarily everything they read online as well. So if there’s just that five-star review, perhaps being a bit more questioning over whether that’s true or not.
Also a lot around honesty with pricing. Consumers are a lot more savvy around “per day” pricing being a bit hiding the truth behind things sometimes, or hidden trial offers, that type of thing. You definitely see that especially come through in user testing and then in the results as well.
I’m not sure I noticed many more changes, but I think the social proof one is the most valuable to understand why that’s happening, because that will inform a lot of other things that we optimizers generally will go to as a go-to kind of winner. We all thought that reviews would hold their truth for a long time, but actually I’ve seen adding reviews to a website lose, even though they’re positive reviews—exactly decrease that KPI. So it’s been an interesting learning seeing that unfold.
Work-Life Balance and Remote Work
Akansha: I also know that you currently work remotely, which is something that even we do as an organization. VWO is a remote-first company. Often times I fail to have that work-life balance because rather than working from home, it feels like you’re living at the office. How do you find that balance?
Steph: That’s a really good question. I think in terms of work-life balance, I actually find it’s a lot better now I’m agency-side, which is interesting. But I think it comes down to the timesheet and the amount of planning that we have. We plan a specific amount of hours for clients, we know how many more hours we have free in the month, so we can look at project work or things that are outside of that. That forward planning gives me a better view of what is going to be happening that month.
Whereas when I was in-house, I found actually it was a lot worse. I was taking work home a lot, even just thinking about it, dreaming about it—it was pretty intense. But that move to forward planning that month has definitely changed my outlook on productivity as well, because you can literally see how much you’ve been doing, which is always really useful.
I guess balancing that now whilst I’m at home—I think I’m very lucky in that we do have flexible hours. As long as we are delivering our work and it’s of high quality and our client hours are being filled, our client needs are being fulfilled, we have flexible Fridays. So I get that work-life balance that day. I make sure I take my dog Loki for a walk, I’ll do the housework so I don’t have to do it on a Saturday or Sunday, so me and my partner have more time together.
I think having that flexi Friday but then also that flexibility within my hours allows me to kind of do those things and have work as a separate thing. It doesn’t work for everyone, but with the timesheet it definitely helps as well because you just can stop and start just as you would.
Challenges as a Woman in Tech
Akansha: The tech industry in general, even conversion rate optimization industry, is a male-dominated sector. In today’s world, if you interview with a tech company, three out of four people who will be interviewing you will be male. How do you tackle these challenges? Making sure that your voice is heard or making sure that your work is seen?
Steph: It’s an interesting one because I don’t think I’ve ever noticed it in black and white—it’s always a subtle thing, isn’t it? I guess generally with optimization you’re quite lucky in that what you’re saying is usually backed by quite a lot of data. So I think I’m always conscious of spelling that out in a very easy-to-understand way, with slides or however they like to communicate, but also ensuring that I have some authority behind my tone.
I’m quite aware of how I communicate, I think, with different people—not necessarily just male or female, but of different positions. Perhaps they need different pieces of information. Some people crave the detail and others just want a short, sharp answer. So I think keeping that in mind and changing my communication style has really helped.
Certainly I learned that when I was in-house and trying to speak to directors or people that were higher up and talk to them about test results and explain stat sig and really trying to go into the detail, and you could see the glaze come over their eyes. Really what they wanted to know was: what money has that made us? And ultimately, finding out how people communicate has really helped.
Other than that, I think you just have to put your voice out there. I know it is scary and you do worry about being shut down or even if you’re a little bit too over-excited sometimes—you’re emotional or something like that. You just have to try and educate as much as you can with people and also just try and push through those boundaries.
It’s really difficult. I think finding women in experimentation can really help. There’s a massive Slack channel now that Luisa started, and that support is there. Women are starting to really be heard in the space, and I think that’s a really positive change to see.
Advice for Newcomers
Akansha: To someone who is just a newcomer who would like to be established in the industry in a way that you have, what would be your suggestions to them—someone who’s fresh out of college with zero professional experience?
Steph: I think the first thing to do would be to just be an absolute sponge when it comes to soaking in experimentation information—really trying to study, reading articles, keeping up with trends, looking at the industry leaders, looking at the likes of Spotify, of Netflix, understanding how they built that experimentation program.
And then number two would be: find a mentor that you can really trust and ask those questions to. If you’re afraid of posting on LinkedIn or going to a Slack channel and saying “I don’t understand this,” then at least with a mentor it’s one-to-one, so it’s kind of out of the public eye and at least you won’t be afraid to ask those questions.
I think that can be really valuable. Certainly I had a mentor at Specsavers and he was really helpful in helping me understand strategy a bit further and trying to hone that skill. But the Slack channels are really useful. Often I’ll post in there asking different questions or answering some people as well if I can. But reaching out to people is probably the best thing, and if you’re not ready to do that, then a mentor is definitely worth a go.
Industry Influences
Akansha: Anybody in specific that you look up to in this field? Someone who motivated you to become a better version of the professional that you are today?
Steph: That’s a good question. I think quite early on in my career I spoke to Tracy—I’m going to butcher her last name—and she interviewed me for Experiment Nation’s podcast. The questions that she was asking me really were thought-provoking, and I remember one of them was “frequentist versus Bayesian.” I remember reading that question and going, “What is she talking about?” And I had to go up to understand how to answer that question and what the difference was, because I was using frequentist but I hadn’t even heard of Bayesian. That just showed how early on in my journey I was.
Actually, she was able to help me explore that, and I think she has been really influential in sharing her knowledge on LinkedIn and being really open with people about her career and how she’s got to where she is. So she’s really inspirational to me.
And then the other person I think that I met along the way when we were in a pitch process at Specsavers is David Mannheim, who used to be at Brain Labs and has now written about personalization. I think the way that he’s built his career and the way he thinks about experimentation but now also personalization is really inspiring. He was just a really nice, helpful guy. He would chat with me on LinkedIn, I’d ask him questions, he was always there to be asked those questions too, and really helped with some career guidance for me as well when I was looking at agency versus in-house. Having those people there that have been through it was just really useful. Both of them I’d definitely recommend giving them a follow.
Statistical Approaches: Bayesian vs Frequentist
Akansha: When you mentioned Bayesian versus frequentist, what’s the one thing that differentiates these in a very simple, layman’s way? We at VWO use the Bayesian model.
Steph: I guess the way that I now try to think about it is that Bayesian uses priors—the prior knowledge. So I think imagine in VWO the tests kind of feed into the other experiments that are alive, whereas frequentist doesn’t. So you’re always just looking at your MDE, your sample size, and then your stopping protocol—whether that’s 90%, 95% stat sig or 80% power—and the runtime that you’re calculating. Then you’re looking to detect that uplift. So that’s the kind of difference.
I prefer frequentist because I feel like I understand it much more in black and white. I do use both, but I think I have more confidence in frequentist. It’s really hard when you’re not a statistician like all of us in experimentation. I imagine there are some that have deep, deep knowledge of the stats and that is amazing, but trying to read these statistical papers when you haven’t really studied that before is really difficult to do.
I think I know that you guys have a few blog articles about it, and I know Optimizely does, and the sequential tool that Georgi has—those are all really useful to look at to try to understand them. But it is difficult, and a lot of the time with Bayesian, people have different opinions as well, particularly around sample size—whether or not it matters seems to be in question quite a lot.
So I think it’s a difficult one, and I just feel like frequentist is a lot more black and white, and I like the way that that works. I use that if I can.
Future of Experimentation
Akansha: How do you think the experimentation space will change over the next few years? Do you see some sort of evolution happening in terms of conversion rate optimization, or do you see the changes will be gradual as it has been in the past? Google Optimize sunset was a huge deal, so do you think something like that would sort of change anything in the future?
Steph: That’s really interesting. I do feel like more and more companies are starting to want to do experimentation. Not all of them obviously have the capacity or the user base to do it, but certainly that mindset is starting to shift. And I think that is going to be really valuable for the industry because ultimately, yes, you can run A/B tests, but also we all talk about the fact that experimentation is a mindset and it’s a valuable business mindset to have. You don’t know what is going to work, so you should try to test it. I think having more companies want to do experimentation is going to help that narrative.
At the same time, we see everyone’s talking about AI, but we’re starting to see how it’s changing things for us at Conversion as well. I’m really excited to see how that unfolds and how far we as an industry are going to take it, because one of the things I wrote about in a recent LinkedIn post is: if we let AI become too close to the experiments and take over too much, we kind of remove the humanity of it.
Realistically, when you’re listening to a user test or you’re watching one or you’re taking a moderated one, it’s listening to people’s voices and understanding their emotion and their face as well as watching their actions. I always wonder how much AI can understand human emotion and actually turn observations to get the true meaning of that insight.
So I think it’ll be interesting to see how that unfolds, and I expect a lot of changes to happen in the next five years with that. The digital age moves at speed, we all know that, so I think a lot will change in the next five years for sure.
Tools and Technology
Akansha: As a consultant, do you rely on a single tool for all your experimentation requirements, or do you like to use a mix of different tools?
Steph: It’s a mix. For the experimentation tool itself, we tend to use what the client has, because I mean they usually have already purchased them, so we learn to use them and we make sure that we can understand the stats model.
And then in terms of housing the experiments themselves and the insights, the observations, all the taxonomy that we talk about with the levers—that’s in Airtable for us. And that has become really powerful. If you look at any experimentation at scale, it’s fed by something like Airtable where you have a clear taxonomy for every experiment. There’s a clear number, a hypothesis, the execution, the risk profile, the build size, all of those things, the outcome, and that can really help improve the strategy.
It’s not just us that have it—we use the lever framework for a lot of things—but I know Spotify have their own that they’ve built out, and that taxonomy sits across their entire experimentation base. I’m sure there are multiple other companies that have it. So I think having that in Airtable, that’s a really good tool.
Other than that, I wouldn’t say there’s any more that are specific to what we do. We tend to be tool-agnostic where we can. We try to use what the client has essentially.
Personal Reflection and Advice
Akansha: Moving away from conversion rate optimization related questions, I do have a very interesting one for you. What is that one advice that you would give to your younger self?
Steph: That’s a good question as well. I’ve thought about this a few times over the years. I think it would need to be something about experimentation strategy, because it’s not really something that is taught. There’s not a course you can take. Yes, you can understand the data points you need, but without having the time and the experience and the learnings from the experiments themselves, I feel like it just doesn’t all come together. You can’t just do a day course and learn experimentation strategy.
So I think it would be something around just trying to take your time with that. Don’t put too much pressure on yourself. Everyone’s making mistakes when they start out, and actually that’s exactly what experimentation is about—it’s about failing fast and learning from it. So I think it would be something along those lines of just take the pressure off and try to enjoy learning whilst you can.
Travel and Competition Plans
Akansha: Do you have any travel plans coming up?
Steph: I do actually. I’m going to Copenhagen in 30 days to compete at Hyrox.
Akansha: My God, that’s amazing!
Steph: Should be good. We’re excited. I compete with my friend and we did London in November, and now we’re training for Copenhagen, hoping for a sub-130 time.
Akansha: How many hours of practice do you get in a day?
Steph: We train three or four times a week, and every session is an hour to two hours. So it’s pretty intense right now. We’re really ramping up because it’s only a few weeks out. Next weekend we’ll run a full Hyrox simulation in our gym to see what time we’ll get, and that will hopefully set the tone for the day because we’re very worried that all this training won’t result in what we want and we’ll be disappointed. But after that, I’m sure we’ll be fine.
Relaxation and Downtime
Akansha: Do you have any place in particular where you’d like to go to relax, maybe a destination or spot that you usually go to destress?
Steph: Yeah, I think it’s usually the beach here. We’re really lucky that the whole of Guernsey is surrounded by beaches, and particularly this time of year the dog can come on them with me. So even a windy walk, just connecting with nature anywhere really, but particularly the beach where I can see the sea—I find it really calming. I’ll take myself there to try and relax, unwind, and remember that there’s a bigger world out there than just the worries in my brain.
Rapid Fire Round
Akansha: This has been great, Steph. I do have a surprise for you—we have a bunch of rapid fire questions and you don’t have time to think about it, so you just get a couple of seconds. Are you ready?
Steph: Ready as I’ll ever be.
Q: Three apps you cannot live without?
A: Instagram, Garmin, and maybe Pinterest.
Q: If a movie were made about your life, what would the title be?
A: “Guernsey Girl.”
Q: One thing that you’d like to change about the CRO industry?
A: A lot of people get bogged down in the debates—the small debates that don’t necessarily change or educate that much. Bayesian versus frequentist is a big one, or should it be CRO, should it be experimentation, should we call it A/B testing or should we call it growth. I think a lot of that—yes, it’s important—but at the same time we’re ignoring a lot of the other important conversations, and it gets repeated over and over.
Q: What’s the most random fact you know by heart?
A: Guernsey is 26.2 miles round—you can run a marathon around it.
Q: Your guilty pleasure TV show or movie?
A: Grey’s Anatomy is the TV show I’ve watched three times through, which is awful.
Q: If you had a podcast, who’s the first person that you would invite on the show?
A: David Attenborough for sure.
Q: Three books that you would recommend to our listeners?
- “How to Kill Your Family”—that’s really good, really enjoying that
- “Hope” by Niall Harbison—he rescued street dogs in Thailand and that was really heartwarming but lovely
- “The Chimp Paradox”—that one’s really good and actually relates to the industry a bit more as well
Q: What is one superpower that you would like to have?
A: To fly. It’s always been to fly.
Q: One thing that you’re tired of explaining to your boss or your client?
A: ROI calculations. They’re just calculations—they’re just our best estimates—and everyone always wants to see them on the bottom line and we can’t control all the outside factors. So that’s a bit of a tiresome conversation for me.
Q: One goal or dream that you would like to achieve in the next three years?
A: There’s too many, but I’d like to run a marathon. I have no idea if my knees are capable of it, so we’ll see. Marathon is 26.2 miles, but only 1% of the population will run one in their lifetime, and I just feel like that’s a really cool stat to be part of. So I really want to run one, but I have historical knee problems. But I would do something like Amsterdam’s because it’s really flat, or London’s because you get to see different scenes. I’d have to train for quite some time, but not for three years, I hope.
Closing
Akansha: Steph, that is everything that I wanted to discuss with you as part of this Women in CRO podcast. Thank you so much for everything that you’ve shared with us. The conversation was so insightful—it was to me, and I know that it will be to our listeners as well.
Steph: Yeah, we’ll do. Thanks so much, it’s been a really nice chat.
Akansha: Wonderful. Thank you so much, Steph. All the best!
Steph: Thanks!
