Editor’s Note: This transcript was created using AI transcription and formatting tools. While we’ve reviewed it for accuracy, some errors may remain. If anything seems unclear, do refer to the episode.
Episode Trailer
Emily Isted: I did this the other day when I was presenting an end-of-quarter report to a client, where you can actually prompt Claude to pull all the data points, every single goal, every single event that we ran in an AB testing campaign in VWO.
And then you can literally prompt it to be like, “Cool, based on all of these experiments we ran, can you please give me an example of five more tests that I could iterate based on these past findings?”
We do find that we’ve got like more impactful test ideas when we’ve got the mixture of the behavioral insights with the quantitative data.
Then the power comes in like super, super strong hypotheses that we actually score in this really structured framework, which allows us to then test the highest impact tests for the, for the business.
So VWO is incredible in the sense that it’s almost like we treat it as a little CRO hub for our clients. You’ve got the AI features. You guys have all kinds of, um, backlog of industry-specific case studies. Obviously, kind of the code editing and the visual editing aspects of VWO are a win as well for us.
Our goal isn’t to kind of, we don’t wanna make these clients dependent on, on us at all, but we wanna kind of shift their mindset to be, “Let’s test everything first, and let’s implement based on the data.”
Guest Introduction
Devon Boyd: Welcome, listeners and viewers, to a new episode of the VWO podcast. Today we’re joined by Emily Isted, Director of CRO at Hype Digital. Very excited to have you on board with us today. Hype Digital is a leading conversion rate optimization agency based in South Africa, but operating globally — combining acquisition, CRO, and analytics to turn traffic into conversions, with a strong focus on understanding user behavior.
In her role, Emily leads experimentation and optimization efforts, working with teams to improve user journeys, usability, and conversion outcomes. At Hype Digital, VWO is part of that experimentation workflow, helping guide decisions with data. Emily also brings a very clear perspective on how marketing is evolving, especially as more journeys shift towards a zero-click world where capturing and sustaining attention is becoming increasingly challenging.
So without further ado, Emily, let’s begin our conversation. It’s going to be an interesting one.
Emily Isted: Thank you so much. Yeah, very excited to be here. Obviously kind of co-hosting with VWO, and yeah, excited to dive in. We’ve got some interesting things to bring to the table today, I think.
Getting to Know Emily
Devon Boyd: Absolutely. Before we do dive in, I just wanted to ask you one or two things to get to know you a bit better and let the audience get to know you a bit better — especially in terms of your CRO journey. Is there a moment or a project that made you just think, yes, this is exactly why I love doing this?
Emily Isted: I think it wasn’t necessarily a project, but it was definitely a moment that clicked into place right at the beginning of my journey — it must have been about four years now. Back when I first joined Hype on the PPC side of the business, we were just starting as a team to build out and focus our attention on CRO and how we could restructure the CRO program and focus on actually bringing CRO to South Africa — because CRO, four years ago, was barely a mention. I mean, we’ve got a little 2% interest now, but it was barely a mention back then.
So I think the shift for me — in terms of yes, this is my calling, this is what I want to be doing — was when that transition happened from the paid side to the CRO side. For me it just makes sense: we research, we hypothesize, we test something, and the data is so black and white. You can’t make up the data. So just to have an absolute answer to a hypothesis that a team has put together — yes, this is the direction, or no, this is not the direction, but let’s iterate because of X, Y, and Z. My brain kind of works like that in general. I like the black and whiteness. So for me it just made sense to dive headfirst into CRO, and here we are today.
Devon Boyd: Nice. I love the decision-making angle that CRO can bring. And I like it as well because after a while it starts seeping into everyday life. It’s like, oh, I don’t know if I should do that — followed by, ah, I’ll just test it.
Emily Isted: I listened to a podcast yesterday evening actually about relationships, and the psychologist was saying, you know, a new relationship — you’re experimenting. Everything in life is an experiment. And then you work through the analysis points and make a decision from there. So yeah, it filters into everything that we do.
Devon Boyd: That’s brilliant. The CRO of dating. The hypothesis — someone says something, and you’re like, hmm, I wonder if I’m gonna test that.
Emily Isted: Just capturing data about people all the time. You’ve gotta draw a bit of a line and remember that we are human.
Devon Boyd: Brilliant. Like real-life session recording. As a kid then, growing up — did you imagine you’d be doing this? Did you think you’d end up in this sort of area, or at what point did the marketing world fall into place?
Emily Isted: No, absolutely did not think I would be directing a CRO team at all. I didn’t even know what CRO was a couple of years ago. I was a performing artist — a professional ballet dancer for years and years. I trained after school to go in that direction and was lucky enough to get a lot of work in South Africa, in films, companies, different dance companies, and that sort of thing. Then I started studying a Bachelor of Business Administration and Marketing online while I was working, just to expand my horizons, because I knew the direction I wanted to head was marketing. I then fell into agency life, was working with a Canadian agency prior, and then actually connected with Cam — who’s the CEO and co-founder at Hype — because we know each other from high school. And that’s how I moved into the CRO space.
Massive career shift a couple of years ago, but honestly haven’t looked back. In terms of the mental challenges and the everyday learnings — keeping up with AI and digital — I don’t like to be bored, and this certainly caters for that. Ticks every box.
Devon Boyd: Yeah, I’m sure. And I think that’s the fun part about CRO in general. When you look at what you’ve studied — the business side — it touches so many different areas inside a business. And especially now, it’s so connected in this omni-channel environment. Users shifting from paid ads to organic search to social to direct visits to AI, zero-click — all of that knowledge comes back and is actually applicable to everything we’re doing in experimentation. How does that shape the way you approach experimentation now, especially the omni-channel piece?
Conversation
The Omni-Channel Approach and Onboarding Clients
Emily Isted: It’s interesting. When we are onboarding a client, we work with clients across all industries. We have almost a two-hour — we call it a spaghetti session or discovery session — where we need to become experts in their specific business and industry in the matter of about two hours. Having that receptiveness and adaptiveness, that willingness to just absorb any sort of information about how these businesses are set up, is super, super helpful.
In terms of the omni-channel approach — depending on which industry these clients are in, and even depending on different businesses in the same industry, our approach to a CRO program or experimentation is always super custom. We are not the kind of agency to follow a blueprint approach. It just doesn’t work. We haven’t even tried it — it’s not something that will ever work for us.
Once we kick off with a client, we do a bunch of research and start running experimentation cycles. It comes down to a quarterly strategic roadmap meeting with the client where we look at their content calendar and understand when are their big sales and big campaigns on the e-com side, or new products being released on the product side, or maybe a new solution on a SaaS client, or a special running for a Safari client where they want users to book between specific dates. We plan with the client according to big events in their calendar, and then we go into a deeper dive into the different traffic source segments. Because naturally we avoid running an experiment that targets all visitors. From the get-go we work in traffic source-based segmentation — a paid experience on the website looks very different to an experience we are sending organic visitors to. We go quite deep from the beginning. We don’t like to waste time on an all-user approach.
Devon Boyd: Great. I love that. And I love the level of detail you go into with your clients. But a two-hour window to try and understand all of that — that’s quite something.
Emily Isted: We had one yesterday. Your brain’s gotta be ready. And then obviously that information, as you start working with the client and the stress tests and all of that — it all just settles in and you become quite proficient in the three months to come. But yeah, it’s quick-moving.
Bridging the Gap Between Data Collection and Action
Devon Boyd: I can imagine. When you’re working with clients that have internal CRO teams already — so they’re relatively mature — do you come across teams that collect the data but don’t know how to act on it? Whereas you can come in and very quickly pull information from here, here, and here, put it together, and be very targeted. How do you overcome that gap?
Emily Isted: At the moment we’ve got a couple of clients that don’t necessarily have internal CRO teams, but they have champions for the CRO work — essentially a head of product or head of UX/UI. They have amazing behavioral tools like the Insights product on VWO, Microsoft Clarity, Contentsquare. They’re gathering all these insights, and then those insights literally sit in a backlog report that goes unopened. We’ve literally seen it.
So how we come in is we bring a structured optimization program — it’s like a simple PXL framework, a test prioritization matrix. It’s a scoring system — I’m sure all the listeners have heard of it, it’s quite universal in the CRO industry. We’ve taken the traditional framework and expanded it. What we do is hypothesize our AB test strategies from these behavioral insights. We find we get more impactful test ideas when we have the mixture of behavioral insights with quantitative data — so GA4 funnel data — combined with qualitative insights too: speaking to the user base, speaking to clients, speaking to customer support teams. The combination of all of that is what produces super, super strong hypotheses that we score in this really structured framework, which allows us to then test the highest impact tests for the business. Honestly, it’s quite simple at the end of the day. That’s all there is to it — that really structured framework.
Devon Boyd: I love it. It is quite simple, actually — though I can tell you it’s not. But it should be in any CRO team or experimentation framework: the hypotheses, the data-backed decision making, the behavioral insights, the quant-to-qual combination. These are so fundamental to running any of these programs.
You’d be shocked — well, I know you’re not shocked — at how many people don’t combine all of those. They skip the hypothesis and almost backwards-set it afterwards. Or they have these amazing tools giving them the insights, but then they don’t use them, as you said — they’re in the backlog somewhere.
Emily Isted: On that point, it is interesting because we are at the point now where we are running quite mature experimentation programs with a lot of our legacy clients since probably the beginning of 2024. These guys are so at one with the experimentation that if someone new joins the team and pitches an experiment that isn’t data-backed, they’re like: hey guys, where’s the data? We wanna see the data. Bring me those three data points. So it’s awesome to see the shift in the mindsets of these clients.
Devon Boyd: That’s so nice. That’s the democratizing of data that comes from running strong CRO principles, which is great.
AI’s Impact on the Agency’s Day-to-Day
Devon Boyd: How are you seeing the impact of AI on that then? Is it adding value to Hype, to your day-to-day, and where is it maybe still falling short for you?
Emily Isted: The hot topic at the moment — AI. We always joke and say we need to actually hire someone in our agency whose full-time job it is to keep up with all these AI releases and actually ensure that we embed those into our systems, because it is a full-time job the way things are moving with it.
But yeah, listen, we’ve taken on and adapted Claude into the business. Our main win around Claude is the MCPs. So at Hype we’ve got three core service offerings: analytics, paid, and CRO. The Claude MCPs allow us to centralize all the data points — Google Ads, Meta Ads from a client, our actual VWO testing and web testing data, as well as behavioral data either from VWO Insights or wherever it may be from. We can centralize this data, and it allows for easier cross-silo work between our PPC team and our CRO team. We’ve all got access to this data at a centralized point, and it can help us strategize based on the cold hard data that we mentioned at the beginning.
A massive pro in the AI direction is the research — honestly, the research synthesis. Instead of taking three weeks to do manual online review scraping for a client and its competitors, we can literally do that with a click of a button now in a matter of minutes or seconds. Little pieces like that have really allowed us to save time. In the beginning we used to have to cross-check a lot of the work to ensure it was accurate, but now the AI is getting better and better.
When it comes to manual scoring — like the PXL framework I mentioned — instead of doing all that manual scoring, AI does that all for us now. Phenomenal, honestly. Probably saved about 15 hours of work time a week with AI usage at the moment.
For falling short — design has been an area we’ve been trying to figure out. We’ve been trying to use various AI tools to push out our CRO test design screens faster. Claude Design has just been released and it’s a really, really powerful tool. But our designer can still manually design what we want faster than Claude can, because you’ve got to have the right design library set up. So right now we’re still happy with our really amazing design team. Design is lacking for sure.
The devs use code-based AI tools a hell of a lot to enhance and speed up their coding. So yeah, honestly, we’re using it on all facets of the business and seeing the adoption with our clients as well, quite fast. As an example, one of our Safari clients is actually incorporating an AI chatbot that is kind of embodying their CEO and basically has the voice of their CEO. Really cool little projects like that happening as well that we are dealing with.
Devon Boyd: Nice. And I love the fact that you’re leveraging the MCP to bring everything into one centralized location — that has so much value. Speaking of chatbots and AI chatbots — did you see the McDonald’s chatbot situation? At one point you could put a normal prompt in there and it would still output that, and then say, would you like a Coke with that?
Emily Isted: Wild.
Devon Boyd: Yeah, as everyone’s getting into the AI piece, things are being rolled out too quickly without being properly quality assured. Quite funny though. Speaking of being able to pull everything into one place and run really cool experiments — with the speed that AI is bringing to that, can you look back and think of any really high-impact experiments, and is there any commonality between them?
High-Impact Experiments and Big Swing Testing
Emily Isted: This is actually a point of discussion in my CRO team at the moment. We’re doing a historical look at the last quarter and really unpacking what’s been working and what hasn’t. It’s really evident that if we are running, say, six small iterative tests on a client, those small iterative tests aren’t doing the job in terms of actual conversion impact. So now we’re changing strategies — even if we’ve got to take 30 hours in a month to focus on building a big swing, something that’s really going to move the needle. Maybe it’s completely reshuffling a navigation menu based on how we know users are actually searching. Obviously all data-backed, but we’re moving from small iterative to massive big swings.
There are probably two or three that come to mind from the last year with massive impacts. The first was an iGaming client, leveraging personalization. When a new user signs up or registers, they get a little bonus overlay and they can select either a sports bonus or a casino-type bonus. Whichever bonus they choose, we would carry forth that bonus messaging in the form of banners across various high-traffic pages on the site. As soon as we had those powerful banners with that first-time deposit bonus messaging, we saw the actual deposit rate increase by 20%. And these guys only care about deposit rates and revenue. So that was really, really massive — five simple banners at high-traffic points in the site, based on personalization. The numbers were outstanding.
Then another interesting one was a local, quite a big travel company where the user was working through an eight-step process to book a flight and had no idea on the control — their original website — how long this journey was going to take. No idea how many steps were in the journey before they could actually have their flight number in front of them. So we added a really nice navigation feature at the top of the website where we had steps one to eight with what each step entailed. This just really added that security and guiding feeling that these users needed to convert. There was a 5% increase in actual flights booked, which translated to honestly over 100,000 rand in additional flights booked. Really, really massive.
Devon Boyd: Fantastic. Two really lovely examples. The breadcrumb trail one — relatively easy to implement, but in terms of that safety feeling it provides. I actually find booking flights myself can lead to all kinds of errors and re-bookings, and that anxiety around double-checking everything.
Emily Isted: Flights these days are such a high-ticket item. You wanna ensure everything’s correct. So you need to have an online experience — mostly on desktop for flights, though people do book on mobile — where the user feels secure and safe, knows what’s coming, can go back, but is guided to the end of the journey. So yeah, it was a really nice win.
Devon Boyd: Absolutely. And with the gaming one — the idea of multi-page funnel testing with consistent messaging following someone through the website. That’s something I feel I don’t see enough of. It’s not just: what is the impact of this page? It’s: what is the impact of this messaging following someone through the whole website so it’s very normalized by the end of their journey?
Emily Isted: And we also take it back to the part of the journey before they even got to the website — integrating and streamlining the ad that they clicked on. Because we also manage the paid side, we have access to control all of that. When that journey is integrated, the results just can’t compare to a journey that isn’t. And it goes into that omni-channel experience track we were talking about at the beginning. Because honestly, we’re at the point now where we can’t really be optimizing pages. You’re optimizing a journey. That’s how we look at it at Hype now. You are optimizing a user journey — working from top to bottom instead of just a static page.
Devon Boyd: Yeah, a hundred percent. I was just imagining your Safari client — I’m scrolling through Instagram, I see really cool imagery for that specific safari, I click the link, I go through to the website, and I want to go specifically to that safari I saw the imagery from. Almost like a deep link into there, where the information is very relevant to where my journey began. And there’s also that additional element of the branding feeling consistent — so I carry the trust from Instagram into the website.
Emily Isted: Exactly. We can be testing at a campaign level and at the website level simultaneously. We can run the same sort of tests by splitting traffic to an exact safari page or to the homepage. And now we work from the top down — at a campaign level. So on Google Ads we’ve got a prospecting-type audience, or let’s work with Meta: a prospecting-type campaign that’s more educational about the company versus an actual conversion campaign that would go to the exact safari page. So we can start splitting traffic at a campaign level as well, which gets really interesting.
Devon Boyd: Oh, that’s really nice. The attribution between offsite and on-site journeys — people so often forget that when someone hits the website or app, they need to continue that experience. The fact that you have control from that top level all the way through is so valuable.
Emily Isted: It fully shapes the paid strategies as well. If we’re running 70% of paid traffic to the homepage and 30% to an exact safari page, we can see immediately what’s working versus not working. And when these guys are launching new types of campaigns, they can take the learnings from previous campaigns and structure the traffic accordingly. It filters into the education level too. You can get into a little bit of a maze, but the learnings are just endless — and that’s what we’re here to do, right?
KPIs, Revenue Per User, and Lifetime Value
Devon Boyd: Oh, it sounds like a great maze. And I really like the idea of pre-segmenting people for their experiments before they even hit the site. In terms of the KPIs the team should be looking at to track real business impact — when it’s tied together in the approach that you’re doing, does that change? Do you look at different metrics?
Emily Isted: That’s where you need to be quite strategic in how you’re presenting data to the client. In the last few months we’ve really migrated towards value of a user. Let me take a step back — it’s obviously industry-based. E-com KPIs and metrics are going to look very different to a lead gen or SaaS client.
What we also do, taking a further step back, is when we present a testing roadmap to the client for the next three months, we like to bucket our experiments into both metrics-based and conversion-based categories. Always ensuring that if a client is interested, we are running engagement-type experiments as well. As an example on the iGaming client — the sticky navigation menu at the top of the screen on mobile would become sticky as the user scrolls, and we’d change the UI to reduce the screen space it takes up and incorporate a little bonus gift icon. That’s an engagement-type test. And we really want to ensure that we’re able to report on as many metrics as possible to the client.
The beauty of these CRO tests is that these clients are having access to data they never had before. On the conversion side, we’ve found that revenue per visitor is kind of the north star metric that you want to be reporting on. With that comes a quarterly forecasted revenue per user metric as well. As long as we are reporting heavily focused on revenue per user and forecasted revenue per user uplift to the clients every three months, and they can see that value, they’re happy.
Now we’ve gone one level deeper and we’re looking at the lifetime value aspect. For a lead gen client, we actually ensure that our testing tool — VWO, Convert, whichever we’re working with — is linked to the CRM via an integration. We link that through, and then we’re able to take the test ID, the user ID, and the experiment and link that up to the actual CRM ID. That allows us to follow through with that lifetime value metric, all the way from ad to website to CRM, and depending on where the lead is sitting in the CRM pipeline. That’s been super powerful — closing that feedback loop in reporting for CRO, and getting really ongoing retention with clients as a result.
Devon Boyd: I like the focus on lifetime value and on the unit economics of revenue per user. Those are really cool metrics for people to be thinking about. Are there any vanity metrics you see people use and you’re like, hmm?
Emily Isted: Not really, because it’s so test-dependent. If it’s an engagement test, then we really want to be tracking and reporting on those specific clicks because the whole point was to increase clicks on the cart, for example. Our focus is always to report on too many metrics rather than too few. I don’t want to be looking at three metrics — I want to have 15 events we’re reporting on. That’s kind of how we look at it.
Using VWO as a CRO Hub
Devon Boyd: Nice. You mentioned a couple of testing tools throughout. I’m going to focus on VWO naturally. I know you use VWO in your experimentation workflow. Has it changed the way that you work with data or insights? And I’ll maybe focus a bit on the fact that VWO has it all under one roof. You’re probably integrating with other tools there — does that impact your workflow? Is it helpful? You mentioned the MCP server.
Emily Isted: Yeah, that’s a big one to unpack. VWO is incredible in the sense that we almost treat it as a little CRO hub for our clients. On VWO you’ve got all the necessary form fields to fill in your hypothesis when you’re hypothesizing for a test, you create your goals and your client-specific goals, and obviously all the events and metrics that you’re going to be tracking. You’ve also got the AI features that allow you to scrape the client’s website to give you some ideas around testing strategies, plus the backlog of industry-specific case studies. So it literally is like a hub of information — a knowledge-sharing, case study information hub. It’s almost such an easy sell to our clients because it kind of gives the user everything in one.
Obviously we use the Insights product for a lot of our clients as well, which allows that integrated behavioral and analysis part of the experimentation to be really, really nicely closely linked to our experiments. So that’s super powerful.
The code editing and visual editing aspects of VWO are a win as well for us. We generally do use just the code base because we’ve got our devs who build quite high-impact tests in the code editor themselves.
The MCP is super powerful in the way it’s changed how we can approach work. I did this the other day when I was presenting an end-of-quarter report to a client — you can actually prompt Claude to pull all the data points, every single goal, every single event that we ran in an AB testing campaign in VWO. It lists out every single goal and lifts out the actual lifts for each event. And then you can literally prompt it: cool, based on all of these experiments we ran, we had five out of ten winners and that resulted in 20% more revenue for the business — now can you please strategize, maybe give me five more tests I could run that are one level up, that I can iterate based on these past findings.
That is super, super powerful for us, because it basically allows us to go forth with an idea that Claude has come up with based on actual test data, and gives us a little kickoff in terms of where our brains can go in terms of strategy. We don’t have to be pulling these strategies ourselves — we’ve got a little kickoff, a strategy idea that we can then run with and build on.
Devon Boyd: Oh, that’s really fun. That’s a really nice way to leverage that MCP server. I’m sure some people are gonna steal that.
Emily Isted: I am not gatekeeping that — people must! It’s phenomenal. And also, the power of that MCP specifically is that when you have the testing product with VWO but you also have the Insights product, it leverages both. It pulls in all that data and combines it. That’s really, really helpful.
Devon Boyd: Fantastic. That sounds awesome. Sounds like we might find a case study in there somewhere.
Emily Isted: Let’s do it.
Zero-Click Environments and AI-Referred Traffic
Devon Boyd: On the topic of AI and zero-click environments, which have become very normal these days — arguably less traffic to websites, but much more focused traffic when it does land. How is that shift in behavior impacting what the CRO team focuses on?
Emily Isted: Interesting one. Our strategies are changing now because of AI. Our Safari client that we’ve been speaking about today — they had one of their biggest bookings this year, in terms of hundreds of thousands of dollars, made from a ChatGPT search. Someone in America had never been to Africa before, prompted ChatGPT for the best safari companies or camps in Africa, our client was pulled into the results, they made a booking, and it was their biggest booking. Huge.
So now our whole strategy for that client has changed. With AI you can’t decide where on the website the AI-referred user is sent to — we have no control over that. But we are able to ensure that we’re setting up the website and the inquiry form well enough to ensure the user can be taken where we need them to go. What I mean by that is ensuring that every single landing page on the website has a nice clear CTA — an Inquire Now button that links the user to the inquiry form. Because we want to be sending AI users straight to the inquiry form. We want them to inquire. We don’t even want them to double-guess.
The other way we’re changing strategies is ensuring that every single page on the website is already fully optimized, because these users are arriving already having decided they potentially want to go ahead. There’s no need for education. We need to get them through the funnel as quickly as possible. A big thing there is your trust and credibility sources — they need to be visible on the page, above the fold. And all SEO practices right now are still important because of AI search. It all ties into one big maze, and we’ve only just arrived at the beginning of it.
Devon Boyd: I love what you said around the education happening almost offsite now. People coming in via that channel — they’re already educated, they’ve done their research via AI to get there. They’re almost a high-trust user.
Emily Isted: It’s been very interesting to watch in Google Analytics — every month the traffic from AI sources is slowly, slowly increasing across clients. That allows us to adapt AI-specific strategies — a completely different landing page for an AI-based user versus a new user. They’re now forming a big part of our segmentation strategies as well.
Devon Boyd: Makes so much sense. You mentioned you can see in Google where people are coming from. Are you seeing differences in behavior depending on whether users come from ChatGPT versus Claude versus somewhere else?
Emily Isted: It’s a bit premature — there hasn’t been much time spent deeply analyzing the different behaviors across the different AI sources yet. But definitely at a high level, different AI sources behave differently. The Claude user tends to be a lot more in-depth — they tend to stay on websites more, which is interesting. ChatGPT seems to be more of a quick research and they don’t really follow through. The Claude user seems to go more in-depth. But honestly, very early days there.
Devon Boyd: Okay. It’s almost confirmation bias on my end, but when I’m using Claude it’s typically for code or when I’m going deep technical on something. When I put a personal query into Claude, it’s normally because I’m being lazy and not switching, because I would switch to ChatGPT for just day-to-day, almost conversational research.
Emily Isted: What I’ve been seeing in myself and in people around me is the adoption — especially in the younger generation — of moving fully away from Googling anything. Now their Google is these AI tools. The landscape is changing fast and furiously, so it’s going to be very interesting to see how that affects conversion at the end of the day. I don’t know if enough of us yet are bringing that AI audience into our segmentation strategies and forecasting. It’s an interesting space right now.
Devon Boyd: Fascinating. And when we talk about building a user profile — you know, meet Mary, Mary is your ideal user — I don’t know if people are talking about Mary being a big user of Claude. I don’t know if we talk about that.
Emily Isted: That’s gonna — I’m definitely bringing that into our user psychology framework tomorrow. Start bringing in the Claude personas. I love that.
Devon Boyd: Brilliant. I think we’ve just coined it.
How Experimentation Evolves Client Mindsets
Devon Boyd: Curious to know — when you’re working with businesses, bringing experimentation in, talking about personas, omni-channel, and conversion across the whole user experience, do you see businesses evolve in their thinking over time?
Emily Isted: We do. When we first start working with clients — speaking to the South African market now, where the CRO industry is less well known — there’s a bit more education that needs to be done for internal teams when we onboard. As soon as we get those initial stress tests out of the way — the tests that come before the actual research is done, just to get quick wins and to run an AA test to check the data is tracking correctly — as soon as those initial quick wins are out of the way, we go forth with the research and dive into bigger strategic tests. And the guys internally really start to see the shift in their mindset.
A lot of people kind of join us because maybe the marketing team or the UX/UI design team really want to push a change live on the website, but the product team is opposing them. The landscape here is very much that businesses then resort to bringing on a CRO team to run experiments — to give company-wide, silo-broken teams data on: yes, we should push this live, versus we shouldn’t. So that’s a big part of it as well.
Devon Boyd: Essentially you’re able to go in and break down silos by using the data — giving teams a way to communicate and stop any politics between them. Not what’s best for product, not what’s best for marketing, but what’s best for the company.
Emily Isted: Yes, and to add to that — when we started with one of our biggest e-comm clients, a clothing company and one of the biggest brick-and-mortar stores in the country, they had recently just tapped into selling online on a Shopify store. When we onboarded them last year, they were honestly only making changes to the website if they had a new campaign going live or if the head of marketing wanted a new homepage design. It was only when we really came in that they got that structure and that testing cadence.
Now they’ve kind of hopped on this testing train, and not only has it been amazing to see the shift in mindsets internally — our goal isn’t to make these clients dependent on us, but we want them to think: let’s test everything first and let’s act based on the data. And as we get deeper and deeper into the tests, the segmentations, and the analysis, the internal guys we’re dealing with — head of product, head of marketing — their minds just roll with ideas. They come to us with more and more ideas each month, based on the learnings they’re getting. Not even just wins, but learnings from losses as well. They get so excited. So yeah, it’s really, really cool to see.
Devon Boyd: I love that level of collaboration. And I like the fact that it’s inspiring them to come to you with even more ideas.
Emily Isted: And the power of these behavioral tools and experimentation to get things over the line in terms of bigger strategic business roadblocks — let me get quite specific with an example. With the same e-comm brand, we were seeing massive drop-off for new users. We were running experiments between new users and returning users, and we were seeing drop-off at checkout for new users five times more than for returning users. The friction at checkout was because a user couldn’t check out without signing up — there was no guest checkout option.
So we built the case for the business based on the behavioral data and the tests we were running, as to why we needed to push this change live. Previously they just didn’t want it live because they actually wanted the data that signup allowed them to access once a user signed up. But when we could prove the value of having guest checkout on the actual purchase conversion rates, they were mind-blown. Little things like that — that’s what makes our work exciting and honestly what gets us up in the morning.
Devon Boyd: Such a nice example. They’re forcing signup, and then it’s like — they just need the guest checkout. But you can still track returning users anyway, so they’re still coming back, still likely converting higher, and maybe then going for the signup.
Emily Isted: We just don’t lose them initially anymore. We were losing 20% of them.
Devon Boyd: Which takes you back round to lifetime value again. One of my pet peeves is newsletter signups — I’m a brand new person exploring your website and as soon as I’ve landed I can’t explore because I have to hit X on your newsletter popup. I don’t know you. I’m not gonna sign up yet.
Emily Isted: We tell our clients: remove that popup. Have it in the footer of the website. People will sign up if they want to. Don’t scare them away. That’s generally the first thing.
Devon Boyd: The only time I’d potentially find it okay — but I’d also say it needs testing — is to have it at checkout: if you sign up, you get 10% off. Though I could also argue against that because you don’t want people sidetracked from the conversion when they’re already in purchasing mode.
Emily Isted: You test.
Devon Boyd: Exactly. It might work perfectly for your audience. I’m glad we’re aligned on the newsletter thing.
Emily Isted: Newsletter alignment. The key takeaway from today.
Shifting Mindsets: Testing to Learn, Not Just to Win
Devon Boyd: If you started working with a new client today, what would be the single biggest mindset or approach shift that changes outcomes?
Emily Isted: We do educate clients on this: we are not testing to win, we are testing to learn. If a test doesn’t win, why didn’t it win? How can we iterate to maybe try and make it win next time? Just embodying that mindset of not testing to win but testing to learn is massive.
And I think another big one for us is that a lot of our clients naturally live, eat, breathe, and sleep this brand — whether it’s their own brand or someone else’s. They’re very attached to it. So with our strategy now of conducting big-impact, big-swing, risky changes that are going to move the needle, they need to trust us. We say to them: we are the experts here. Based on all the research we’ve done, this is what we propose, and it’s going to move the needle for you. And remember — we are not designing and building this test to be implemented in the live environment. We are designing and building it to run in a test environment with real users on your site or platform, because we can only really learn if we are testing in a real environment.
So we say to them: if you’re feeling a little bit scared or it feels risky, let’s run the traffic split 80/20 — 80 to the original and 20 to the test. We work with them to get to where we need them to be in terms of risk tolerance. Then we guide them through: look at all this data coming to the variant, we’re making you more money, let’s increase the traffic split. So it’s just about changing the mindset from: we are not pushing anything live, we are testing, and it’s only going to benefit you at the end of the day.
Devon Boyd: I love that. It’s such a strong message. And even that conversation about risk tolerance — what is your risk tolerance? Are we going in fast for velocity and learning so we can keep iterating, or are we taking a more cautious approach? Both work and both can be very specific to the industry.
Emily Isted: I don’t think we’ve actually ever asked a client: what is your risk tolerance? I’m literally going to ask them that. That’s going to be my next client meeting question.
Devon Boyd: It’s a nice one. I use it when talking to low-traffic sites, around statistical significance. A lot of people quote the 95% without knowing where it comes from. Sometimes for a low-traffic site, we might want to lower what our significance threshold needs to be. That’s again — tolerance for risk.
Emily Isted: Exactly. We can take in less data and move on.
Devon Boyd: Yes. Because Mary, our ideal marketing profile — she uses Claude, but she’s got a great tolerance for risk. Actually she hates newsletters too. Don’t go there.
Rapid Fire Round
Devon Boyd: So Emily, we’re towards the end of our time together. Some rapid fire questions for you — some funny, some not so funny.
Devon Boyd: One CRO metric you wish people would just stop obsessing over.
Emily Isted: That’s a tricky one. Honestly — bounce rate, engagement time on page, time spent on page… hmm. I’m not actually going to answer that. I like all metrics.
Devon Boyd: That’s fair. I’ll give mine though — flat revenue, the hard conversion flat revenue number, especially in e-commerce. It doesn’t always capture returns. People don’t factor that in. So it’s important to look at, but when people obsess over it I always think: are we looking at the right metric at first conversion, or do we actually need to capture some later data?
Emily Isted: I’d add on that — overall conversion rate, whether it’s a purchase or a lead. It’s epic to show the conversion rate lift in the test, but there are so many environmental factors at play externally when it comes to actually monitoring that conversion rate once a change is implemented. Very fickle.
Devon Boyd: One thing your non-industry friends still don’t understand about your job.
Emily Isted: I was sitting next to some friends at a co-working space a couple of weeks ago and they said: Emily, we have no idea what you do, but you sound like you’re very good at your job. And I said: thanks guys. I’m not gonna explain it.
Devon Boyd: I’ve tried so many times and I think I make it so simple, but people just go: I know you’ve explained it and at the time I thought I got it, but I can’t get it now.
Emily Isted: Try explaining an A variant and a B variant on a website to the parents’ generation. No hope.
Devon Boyd: I had a breakfast at a B&B not too long ago and I was talking to a physiotherapist. I tried to explain what I do and he just said: it’s not very physical, is it? You can’t really touch it.
Emily Isted: I did connect with my physio the other day on AI though — they’re really leveraging AI in the practice with transcription of notes and emails to clients. I was like, wow, you’re a really new age physio.
Devon Boyd: That’s really cool. I was going to ask what one thing AI will probably take over in the next two years — but apparently it’s physiotherapy.
Emily Isted: There we go. Um, AI overtaking things — honestly, never the people. We are always going to need the strategy. In our line of work, we’ll always need humans. We’ll always need client interactions, the creativity, the strategy.
Devon Boyd: Creativity as well — I always think of AI as backwards processing. It can be trained on everything up to right now, and it can predict and suggest, but the suggestions are always trained on things that already exist. Humans can come up with something weird or wacky until it’s not weird anymore. I always think of Picasso — when he started doing his thing, people probably went: that’s strange. I don’t think AI could ever have come up with what Picasso did.
Emily Isted: It was weird.
Devon Boyd: Exactly. If you weren’t in CRO or ballet, what would you be doing now?
Emily Isted: Maybe running a safari camp, honestly. A guide out in nature. Outside.
Devon Boyd: Not yet. But you can tag your Safari client and say, by the way…
Emily Isted: She’s already said if I want a partner discount, let me know. So that’s exciting. But yeah — something outdoors.
Devon Boyd: For anyone who’s looking to start a career in CRO today, what advice would you give them, or what is one thing you would have done differently moving into it?
Emily Isted: In South Africa we have to hire people in the CRO space who maybe aren’t that experienced, because we don’t have many experienced CRO professionals here. So when hiring locally, we have to hire people who are very strategic, very curious, and able to learn quickly. Because what we do is learnable, if you have enough curiosity and creativity.
Someone interested in CRO needs to be quite full-bodied in terms of strategic mindset — super organized, great with clients and communications, great with internal teams. You’re doing a little bit of everything: project management, account management, a bit of ops. So in terms of a career direction, it’s quite a strong and exciting one.
Devon Boyd: And it’s always changing. In CRO, your operating space is the internet, so you are always at the forefront of it. Like having to learn how to use AI in all your workflows — you are always there.
Emily Isted: When cookies change, or when we have to move from third-party to first-party cookies — we are led by what’s happening on the internet. So being super adaptable is key. Someone who doesn’t like change would not be a good fit. Someone adaptable, open-minded, and curious.
Closing
Devon Boyd: That’s really nice advice. Any final thoughts or messages you’d like to share with the audience before we conclude?
Emily Isted: I really want to — and it’s on my list for the rest of this year — just knowledge share. Connect with people in your space as much as you can. Pop someone on LinkedIn who you really enjoy, whose content resonates, and pop them a message. Set up a 15-minute chat. I’ve done that in the past with a couple of CRO business owners and some big names in the CRO space. It’s been amazing to see the camaraderie in the industry and how open people are to sharing their learnings, what’s working, what’s not working. So yeah — just keep sharing, keep connecting with each other wherever you can.
Devon Boyd: I like that. The idea of give, and naturally you’ll receive. Emily, thank you so much for joining us today. It’s been really fun to talk with you and go through your experiences — how Hype Digital is pushing everything through your workflows, how you’re connecting AI, and how you’re adapting to the world of AI. It’s been great to have you on the podcast.
To all of our listeners and viewers, thank you as well. Thank you for your time and for listening to what Emily had to share with us today. If you enjoyed the episode, be sure to subscribe and stay connected for more conversations like this one. Thanks everyone.
