Editor’s Note: This transcript was created using AI transcription and formatting tools. While we’ve reviewed it for accuracy, some errors may remain. If anything seems unclear, do refer to the episode.
Trailer
I love how varied it is. I love how no day is the same, no week is the same, no test is the same. You get to flex your analytical, quantitative, statistical muscles as much as you get to use your design brain and your UX side of thinking. It really ticks a lot of boxes.
CRO is not just the testing side of things — it encompasses a lot of research, and understanding your customer is as important, if not more important, to lay a good foundation for the tests and experiments you’re going to run.
A lot of times, your primary metric in a test is a quantity-based metric — click-through rate, add-to-carts, interactions with a function or feature. It’s really valuable to think of the comparable quality metric for that quantity — your progression to PDPs, your engagement post-search. But even if you don’t see an outright winner — one you’ll implement onto the website — the learnings involved in that analysis are equally as valuable, and how they’re applied after that test matters.
Customers are becoming more and more brand-agnostic in the way that they shop — the level of brand loyalty we’ve seen historically is moving away a little bit. CRO can become very powerful to a business, using it to understand why people shop with you — exactly the experiences people are looking for, and the experiences that might hinder someone.
Sometimes, in situations where something doesn’t work, this is where CRO can be really powerful — it makes you question your own assumptions. You might go into a test with a hypothesis like “there’s no way this can fail, this is much better” — and it doesn’t. Well, then you’ve learned something.
About VWO
Welcome to another episode of the CRO Wizards series by the VWO podcast. In this series, we speak to top CRO leaders in e-commerce, media, subscription, retail, banking, and other industries about CRO strategies and the positive impact they can have on your business.
Before we speak to our special guest for this episode, here’s a quick summary of who we are and what we do. VWO is a leading experience optimization platform. Using our latest product, VWO Insights, you can analyze user journeys and identify conversion roadblocks on your website and mobile app.
So, without any further delay, let’s jump right into the conversation. Today we are delighted to welcome Emma Orton, a leading CRO specialist at Super Retail Group. Super Retail Group is a powerhouse in the retail world, proudly managing four iconic brands — Supercheap Auto, Rebel, BCF, and Macpac — as one of the largest retailers in Australia and New Zealand. Super Retail Group is at the forefront of innovation in the retail industry. Join us as Emma shares her expertise and insights into the world of conversion rate optimization and how she drives success for such a prominent retail group.
Introduction
Jinal: Hello, Emma! Welcome to the CRO Wizards series by the VWO podcast.
Emma: Hello, thanks for having me.
Jinal: So tell us, is there something interesting you’re working on this week, or your plan for next week?
Emma: At the moment, we’re working on planning out the rest of our financial year, and putting in some exciting things for the future — just figuring out what we want to be doing.
Jinal: Growing up, did you know you wanted to be in this field, or was there something else you had planned or dreamt of doing?
Emma: No, not at all — I didn’t even really know this was a job for a long time. At one point I actually wanted to be a vet, and then that quickly dissipated. I always kind of wanted to do things with people. I was interested in law, or linguistics, or psychology. So this sort of feels like the digital application of that, but no, I didn’t even know this was a possibility until, you know, a long time later.
Jinal: Interesting — it’s interconnected with a sense of psychology, and that also comes through. But yes, as you mentioned, it’s in the digital space.
Jinal: And what do you love most about the CRO industry?
Emma: I love how varied it is. I love how no day is the same, no week is the same, no test is the same. You get to flex your analytical, quantitative, statistical muscles as much as you get to use your design brain and your UX side of thinking — really delving into people and their psychology and how they make decisions. So it really ticks a lot of boxes. For me, I love the variety.
Jinal: So it’s like you fly with multiple wings into this world — what’s going to be required of me today?
Emma: Yes, exactly.
Jinal: Now let’s understand — how important is it to incorporate the voice of the customer into your experimentation strategy, and how do you go about it?
Emma: Yeah, absolutely — it’s really important. We have it as a staple part of our ongoing research. CRO isn’t just the testing side of things; it encompasses a lot of research, and understanding your customer is as important — if not more important — to laying a good foundation for the tests and experiments you’re going to run. You need to be able to research and understand, to identify the issues, so that you’re running an experimentation program that is data-driven and issue-driven. But you also need to understand your customers — what do they want, where are their specific pain points — and voice-of-customer insight, whatever medium that may be, whether it’s a survey, feedback tools, or product reviews, all factor in to making one big, holistic, detailed understanding of what’s going on.
So in addition to the research phase being really important, it’s important to make it insight-driven for your tests. We have an always-on voice-of-customer survey running at all times — insights from focus groups, and even blending in offsite and in-store data, understanding how people are using our web chat functions, and the things they come to the contact center with. These are all fantastic, rich resources. Layering that on-the-ground, verbatim feedback with what you know about your website performance gives a really rich tapestry to understand what’s going on.
Jinal: That’s interesting — you also consider offline data, like store data, to understand how offline sales happen versus how people perceive the same bouquet of offerings on the website. So that’s a nice blend of data you bring in when analyzing customer insights.
Emma: Yeah, I think we’re fortunate that with such a big business, we’re not short of customer insights. So it’s about harnessing them, knowing where they are, and using them to layer on top of the quantitative side of things especially.
Jinal: Experiments are typically focused on improving a primary metric, or maybe a couple of secondary metrics. What are some data points that CRO experts often miss but should consider while analyzing test results?
Emma: I think it’s helpful to think of the metrics you’re measuring in a test as categorized into quantity-based or quality-based. A lot of times, the primary metric in a test is a quantity-based metric — click-through rate, add-to-carts, interactions with a function or feature. That’s the thing you’ve changed, and you want to measure whether people are using it more.
When we think about the overall experience effect that your test is having on a user on the website, it’s really valuable to think of the comparable quality metric for that quantity. So not just interactions with the search function, for example, as your quantity metric, but also — has the overall experience improved? — as your quality metric. That would be things like progression to PDPs, engagement post-search, and so on. So I think a truly successful test would show success in both its quantity and its quality metrics, and I think those bind together well.
Jinal: And that helps carry the success forward. Let’s say the primary metric looks great, but the secondary supporting metrics aren’t as green — that can create some confusion about whether to call it a winner.
Emma: Yeah, you can send yourself down plenty of rabbit holes like this, but I think it all feeds into that understanding of how your customer has behaved. So even if you don’t see a straight “yes, this is an outright winner, we will implement this on the website,” the learnings from that analysis are equally as valuable, and how they’re applied after the test matters. The metrics of a test don’t stop when you hand the report to your stakeholders — you should use them to inform what you do next.
Jinal: That gives depth from a data lens as well — rather than relying on just one metric, you consider what else could have been factored in. So, what are some common A/B testing mistakes that should be avoided, since they can lead to invalid results?
Emma: I have a couple, and they all boil down to a general theme: running experiments that don’t have enough statistical power to determine a result. You risk a false positive or a false negative when interpreting the results. A key component of testing is obviously sample size — you need a sample size large enough to detect differences outside of chance. We all know about statistical significance; that’s a key component. But you also need a data set large enough to give an accurate representation of your audience.
Running a test that doesn’t continue through the weekend, for example — the types of customers you have Monday to Friday might differ from your weekend customers. So giving tests long enough to run increases your statistical power, increases your sample size, and gives a much more reliable indication that your sampled users are representative of your general customer base.
Jinal: You mentioned statistical power — is there a benchmark you follow, like anything above 80% is okay to move forward with, and anything under should be discarded? Or do you rely more on case-by-case judgment?
Emma: A bit of both — we take it case by case. Our gold standard is 95% statistical significance; that’s our “yes, let’s proceed without question” threshold. Not every test reaches significance — in fact, most tests fall just shy of it, though you could argue the case that the result is still positive even without hitting 95%.
We’re pretty stringent on the 95% mark, but anything less than that by no means signals failure. Like I mentioned, we take those results and dig into what went wrong, how we can enhance the approach in the future, or we simply consider that not every change makes a material difference. In a case like your 80% example, we’d look at the trend — it may not have reached 95% statistical significance, but if it’s been fairly consistent throughout, we can be comfortable that it’s a low-risk change to make, and move in that direction if the business wants to.
Jinal: And the gap between the control and variant graphs matters too — the higher the confidence, the happier we are. If the lines overlap less, that’s a good sign of performance. And even if it doesn’t reach the 95% benchmark, at least we know we were on the right track, even if we couldn’t quite get there.
Emma: Yeah, it’s all in the name of proving or disproving your own hypothesis, and that doesn’t always have to mean “we make a change to the website.” It can just be a continuous hypothesis-learning process. I think we also need an understanding that a test can lose, and it can win — both are two sides of the same coin.
Jinal: So, Emma, when you present research findings or a test proposal to a stakeholder, how can it be done more effectively to secure stakeholder buy-in?
Emma: I think buy-in is crucial if you’re going to progress your programming and really move forward to achieve the outcomes you want. One thing I’ve learned and observed at Super Retail Group, particularly in an in-house CRO position, is that CRO really becomes a powerful tool when it’s used alongside shared business goals and strategies. When it’s used in tandem with projects and initiatives from stakeholders, you address the buy-in from the start — you’re going in with shared context.
It’s important that your stakeholders understand, before you propose a test or piece of research, the power CRO can bring — the value that research and testing can bring in reducing risk to those plans and initiatives, and giving more insight into revenue impact and whether something will work if progressed.
When it comes to actually proposing a test or research project, the simpler the better. I think telling a story is a really effective way to do it. It’s great to have your data behind you and to be research-driven in your explanation, but keep it focused on the effects and impacts that matter — you tend to lose people if you go too heavy on the numbers. Bringing it back to tangible impacts that people can relate to makes it really powerful. Treating it like a story — “this is happening, we will do this, and it will help in X and Y” — is a clear way to explain yourself and maintain that buy-in from stakeholders to encourage progression.
Jinal: It’s like data is our soldier heading into the field when presenting ideas — and it’s also important to understand how a stakeholder might react. If it’s a “no,” it’s important to understand why — is it brand guidelines, past learnings, or something else restricting the path forward?
Jinal: So tell us, how do you conduct a competitive analysis to identify opportunities for optimization?
Emma: Competitor analysis is a key component of any research process when it comes to test ideas and opportunities to enhance the experience. We might take a look at the competitor landscape and the competitor journey across all the major touchpoints we’re analyzing — for example, if we’re looking at search, we’d look at all the search-based touchpoints across the site.
One key component that’s crucial in competitor analysis especially is that it can fall into the realm of being subjective — you might look at a website and think something different than I do. So implementing some sort of standardized scale, and bringing qualitative observational insights into a rank-based formula with attributed scores, helps add a layer of quantitative rigor to something that’s otherwise pretty subjective.
The real power in bringing competitor insights to understand where to go next is in layering them alongside your own knowledge of your customers. It’s important to remember that you don’t have the business context of that other retailer to explain why they made certain decisions — but you do have the context of your own business. So while you might see a great function and not know why they built it or how it’s performing, you can ask whether it addresses a design pattern that solves a specific problem you have. It’s the combination of understanding what’s happening in the market — the wins and the losses — with what you already know about your customers and your website.
Jinal: So it’s also a blend of past experience — knowing how users these days are adaptive to certain elements. We might suggest something should be there, even though we haven’t tried it and don’t have the data for it, because we have confidence based on past experimentation and learning trends from market research as well.
Jinal: How do you apply that layering — blending competitive insights with your own understanding?
Emma: If you conduct your own internal research on your website — say, on search, in this example — you understand your own search experience, your wins and losses. You can map out a funnel, understand where people are engaging, whether they’re using predictive search, what their search terms are, and so on. So you know a lot about how people use your search, and you have an idea, just based on your own data, of where you might find opportunities or issues that feed into testing or future enhancements.
If you then compare that to a competitive analysis and see a competitor using predictive search in a way that highlights a gap you’d already identified on your own site, those two things together form a tangible option for what you may do.
Jinal: That’s a wonderful way to explain it — we have our own problems, we identify if something is being done better elsewhere, and then we bring that forward as the reason for pursuing a competitive-informed approach, with confidence.
Emma: The great thing about competitive analysis in the CRO space, alongside testing, is that it’s something to be tested. You might see an amazing design that fits your problem perfectly — great, we’ll use it as potential variation inspiration — and it might still perform terribly. But at least you’ve learned from it and mitigated the risk of just implementing something because a competitor is doing it. That’s why you need a testing layer as part of the whole process.
Jinal: Right — it might be terrible, but then we understand whether that’s because our problems are different, or maybe our audiences are different too.
Jinal: Given your background in psychology, how has it helped you become better at CRO?
Emma: Massively. I mentioned I didn’t even know this was a job, but it feels like a perfect fit now that I’m in it. The most practical, day-to-day answer is in research methods and experimentation methods — my whole degree was about understanding different research methodologies and conducting certain types of analysis, all to create a hypothesis and run experiments, along with the protocol and process of that experimentation and the analysis on the back end. My day-to-day is conducting research and running experiments, so the two are very interlinked, and the skills I learned throughout psychology were absolutely transferable in that practical application.
There’s also a level of — I did psychology and ended up in this world, in the CRO space, because I’m interested in people. I like and care about how people make decisions and interact with an interface, and how you can make that experience better — and if the experience is better, people ultimately progress further into the journey you want them to take. How people emotionally interact with brands really interests me, and that passion is what got me into this space and keeps me engaged. It’s not a chore for me to constantly learn and read and listen to things around this space — I’m just interested in it.
Jinal: I think that brings more energy to your current practice — your background just adds to it. It’s interesting to hear how it’s helped, because not everyone would say their background helped this directly. Yours is a real reflection of how previous experience can strengthen this practice.
Emma: Yeah, and I think the great thing about the digital space in general is that there are such varied backgrounds among the people who work in these positions, and I think that variety and range of thought — how people approach things and what they’re interested in — only makes everything better.
Jinal: So tell us, what major hurdles do you think retailers will encounter in the near future, and how can conversion rate optimization help them?
Emma: I think a big one is that customers are becoming more and more brand-agnostic in how they shop — the level of brand loyalty we’ve seen historically is moving away a little bit. That’s not to say customers aren’t using CRO as a driver anymore, but the market is becoming much more competitive, especially in the digital space. So CRO can become very powerful for a business, in terms of regularly testing and validating your own assumptions about your customers — using it to understand why people shop with you, exactly what experiences people are looking for, and what experiences might hinder someone from continuing to progress. Using CRO-based insights from testing and research to lean into why customers shop with you, and using that understanding to enhance your own experience, is ultimately what’s going to make a difference in a competitive market.
Jinal: Any piece of advice you’d want to give to retailers who are new to CRO, or any wisdom you’d want to share?
Emma: Just try. I think you start with what you’re curious about and what you want to know about your customers. Find that out, whether it’s through Google Analytics, a short survey, product reviews, or feedback on your website. Understand what you’ve got now, start to interrogate that data a little, and see what naturally comes from it. Understand what your hypothesis might be to improve on the issues you’ve highlighted, and just give it a go — do a test and learn from there. The best way to learn is by doing it. And it’s not testing exclusively — research is equally as powerful in arming people to make the right decisions. So start with what you’ve got, and just give it a go.
Jinal: That’s a great motivation — trying is more important than building backlogs. There should be a day where we actually try what’s in the backlog and see how it helped, or if it didn’t work, that becomes even more motivation to keep trying.
Emma: Yeah — use your test results as an additional piece of data. It’s okay if it didn’t work; ask what hypothesis can come from something that was inconclusive or negative. It becomes critical, and you can use that to move forward.
Jinal: It makes our brains exercise more when something doesn’t work — you think, “I could have tried this or that,” and the design rethinking happens from there, because now you know this approach doesn’t perform well, and the next iteration might be one of the last chances you get to try.
Emma: And sometimes, in situations where something doesn’t work, that’s where CRO can be really powerful — it makes you question your own assumptions. You might go into a test with a hypothesis like “there’s no way this can fail, this is much better” — and it doesn’t work. Well, then you’ve learned something about your customers, and the business has taken risk mitigation seriously, and hasn’t proceeded with something that was ultimately incorrect. I think a lot of the benefit, in addition to increasing growth in revenue and other big business improvements from enhancements to the online space, is that you grow your customer understanding and question your own assumptions.
Jinal: And that’s the reason we test rather than directly implement — even when it doesn’t work, we’ve prevented revenue loss. Had we not tested and just implemented, we wouldn’t have known.
Jinal: Tell us, how are you using AI tools in your work, and do you believe AI will help improve optimization efficiency?
Emma: At the moment, I’m using AI primarily in the data-grooming and admin side of things — Excel formulas, cleaning up spreadsheets, that sort of stuff. I think a lot of the efficiency that can be gained through AI, and I certainly think there’s space for AI within the optimization industry, is in that data and admin space — taking away the menial, repetitive, low-impact tasks like data entry and spreadsheet grooming, and taking that off a strategist’s plate so they have more time to dig into the data and understand their customers. That will increase the outcomes of that research and understanding.
Jinal: Do you foresee it helping others in the future too — those who are new, or those willing to use AI in optimization, maybe in copywriting? I know copywriting needs creative thinking, but perhaps those who haven’t started can give it a shot to save time and be more creative.
Emma: Yeah, for sure. I think there are certain tasks where, of course, you can use it, but I’d encourage anyone fresh in the industry to first do things without AI. I think that human-level understanding and critical thinking applied to what you see is important, because our customers are people, and people don’t necessarily behave as you’d expect, or perfectly rationally. Having that lens — being able to see outside the norm — is really crucial. The creative element is a big part of CRO, in terms of formulating and thinking about variation designs and test hypotheses. I think keeping that human is important.
But the AI side of things is great for things like attaching quantitative figures to competitor insights, for example — that kind of categorization work can absolutely be taken off someone’s plate through AI.
Jinal: So it becomes an assistant to us, but not something we entirely offload our thinking to — more like a trigger to point our minds in a direction.
Emma: Absolutely, and you’re right in that framing — AI essentially becomes an extension of the team, like an additional member. I think there’s a level of peer review needed for anything that comes from AI, just like we check each other’s work — let’s also make sure we’re checking the AI’s work as well.
Jinal: Any final thoughts you’d want to share before we conclude?
Emma: Just to encourage people to pay attention to the customer insights they have. There’s an ample amount of information you can get from customers online, whether it’s straight quantitative data or just from experiencing your own website as a customer, or doing a walkthrough. Take the time to do the research at whatever stage you’re already at, and use those insights to think about what you might do next — and encourage people to start testing some of the next steps that come out of the data.
Jinal: That’s a great piece of motivation for all of us.
Emma: I think the headline would be — encourage the learning. A lot of it stems from the depth of understanding you have, and the quality of your research. The knowing is so powerful in terms of what the solutions could be. Learning is a power — the more we learn, the more we build our own data bank, not just for testing but for features and everything else. If we stop learning, that’s where we stop thinking about what’s next.
Jinal: Absolutely — otherwise we look for easy references available in the market instead. The more we focus on learning, the better we’ll be able to know how results might perform, or how they should be curated to our audience segment.
Rapid Fire Round
Jinal: If you were starting a career in CRO today, what’s one thing you’d do differently?
Emma: I don’t think I’d do much differently, but thinking back to my earlier days — when I was starting in SEO and the paid ad space — I’d probably apply a test-and-learn process more rigidly. I was doing a bit of it ad hoc, but I’d lean into the principles of experimentation earlier than when I actually started in CRO.
Jinal: Who do you look forward to in the CRO space — any personality you follow or admire?
Emma: I’m really enjoying what David Mannheim is doing with his intent-based venture. I think it’s awesome — he’s got a good lens and thought process around personalization specifically, and customer understanding.
Jinal: Hope David is listening — we’re his followers too! Any book you’d recommend to others?
Emma: I’ve read a few — books that are sort of adjacent, even if I’m reading a business-based book. One I read recently that I really liked, around human-centered design, was I’m Afraid Debbie from Marketing Is Away for the Day — a really interesting look at practical applications of human-centered design and testing in office spaces and real life.
Jinal: What’s your go-to travel destination in Australia?
Emma: I went to Western Australia on a road trip for the first time last year, and I’d just want to go back. I’m definitely a WA convert.
Jinal: One thing you think AI will replace in the next three years?
Emma: I hope data entry — at least from my perspective!
Jinal: If not a CRO specialist, what other profession would you have chosen?
Emma: It’s difficult, because I was never really sure until I discovered this was a job. When I left university, the other path I was very close to taking would have been going down the law route, doing a law conversion.
Jinal: And psychology — would you want to go back to that space, or take it forward differently?
Emma: No — I really liked it as my degree, but I realized pretty quickly I didn’t want to go clinical with it. Where I found a lot of my interest and passion was in its application in other areas — this being a prime example, but also its applications in law, business, and economics in general, and obviously the digital space.
Jinal: One metric you wish people would stop obsessing over?
Emma: Can I say conversion rate? Not to do myself out of an acronym, but there are so many factors that go into an overall conversion rate — if we’re going to hold that up, let’s give equal weight to everything that goes into it. All the micro-steps leading up to it are equally, if not more, important to understanding the experience of your customers than just the end conversion rate.
Jinal: Get specific — that’s where a lot of the value comes from: exactly what behavior are you interested in? Multiple behaviors fit together to make up your conversion rate, but if we can pull apart the dials on one, what is that behavior, and what’s the corresponding metric?
Jinal: A dream or goal you want to achieve in the next three years?
Emma: Professionally, to grow the space at SRG — keep the momentum going and further embed experimentation as a core business function. That would be awesome.
Jinal: And personal goals?
Emma: Travel more, for sure. I’m also trying to buy a house — that seems to be the never-ending timeline of a goal!
Jinal: Hopefully you achieve both — travel and growing the CRO space.
Emma: I’ll try!
Jinal: All right, thanks for an insightful conversation today, Emma. Stay tuned everybody — happy optimizing. See you!
