Front-End Development For A/B Testing: How To Get It Right

Welcome, Johnny from journey further. Thank you very much. Thanks for having me. Great. Before starting with the actual discussion, I want to let attendees know that you too can participate in this discussion.

Go to webinar does not allow me to switch on your camera, but I can switch on your mic’s. So do share your thoughts on the questions being discussed.

Send me a request using the chat or the question box from the control panel, and I will be happy to unmute you. Johnny, please take it away. Great. Thank you very much. Okay. Hello, everybody. Thanks for joining the call.

Just a bit of an introduction to me first of all. I’m currently the digital experience director of a agency called Journey Further. We’re a performance marketing agency, so we do paid media, organic, and digital experience.

Digital experience mainly means fully retained conversion optimization, but we also sell project work and consultancy as well. So I do quite a bit of management consultancy.

I have a long background in conversion optimization. I started doing it around 15 years ago now when it was first invented really, and that’s pretty much all I’ve done since both clients side and agency side.

I’ve worked with the likes of NIKE o 2 and right over the very big companies and I actually built and ran a big in house experimentation function at Skype embedded into all their product functionality in the UK.

And so that’s me. What we’re going to talk about today is front end development in AB testing. So Like I say, I’ve been in sort of all walks of life around experimentation.

I’ve been in house on primarily, nearly on my own doing stuff. I’ve worked in agencies. I’ve been in big organizations with huge numbers of software engineers and things like that.

And I think the consultancy work that I now do, I get exposed to a lot of different kinds of companies at different stages of their development.

And the 1 thing that always strikes me is that there’s very few people getting the front end development out of it right.

So we’re gonna talk about the different types of front end development that can be used within AB testing and and how AB testing delivers value because What I’ll talk about there is how an important part of AB testing is efficiency.

The efficiency of testing things and that means that for an end development has to be efficient.

And so where typically people go wrong and have to get it right? And then we should have some time for questions at the end. So the types of front end development for AV testing.

Well, first of all, you tend to find this sort of evolution that people go through which is not necessarily always linear, but you tend to find this evolution where right at the beginning you’ve got usually an in house person who probably doesn’t it’s not their whole job to do conversion optimization.

Trying to do some AB testing on a website, and they’ll have an AB testing tool, and they will probably be trying to build tests using an editor, a WYSIWigg editor. So for anybody in washington, it doesn’t know where that is.

AV testing tools almost always, have a visual editor that allows you to try and build tests without using any kind of code, so dragging and dropping things, changing copy and things like that.

They’re very limited in terms of what they can do, but they’re a great place to start.

So right at the beginning, you tend to find people using those and then As I said, they are very limited, so it’s not very long before whoever is doing that realizes that they are not gonna get very far using a wizard.

We get So they want some development resource, and the first thing that usually happens is they’ll find someone with some coding skills from somewhere you know, maybe somebody in a different team who can help them or somebody in an agency or something like that who can help them.

And and we’ll, you know, try and scrape a bit of favors of people who can know a bit of code. And from there, you tend to get a bit more formalized version of that where there are there is some allocated support.

So if you’re an in house conversion optimization practitioner, you might have a web development agency or an in house engineering team who actually are managing and hosting and monitoring the web site and doing web development and maybe you get a little bit of time allocated from that kind of development resource.

And so you’ve got some sort of formal number of hours a week from a developer who can help you use that.

And on from there, you tend to get more of a dedicated but so, you know, a conversion optimizations team starts to form and they might get their own developer within the team.

And then from there, you sort of start to expand into into product.

So bigger organizations tend to be organized into product teams and then there’s 2 different types of those like, you know, initially you’ve got product teams where there is a some sort of centralized experimentation resource.

So you know, people who are running tests are trying to influence their product teams to do testing.

And the the the holy grail in a way is where you’ve got a center of excellence model where you’ve got product teams running tests, but you’ve got a center of excellence that acts as a kind of governance So there’s lots of different ways that AB testing can be done in organizations, and all those have very different approaches to front end development.

But the key thing that I want to communicate with all of this is it does not matter how advanced businesses are in this respect, they can still be getting it very, very wrong.

And believe me, I’ve seen it. So like I say, I do quite a bit of management consultancy. I work with some quite a very big companies around the world who have big product organization structures, and they’re not getting it right.

And so it doesn’t matter It’s not a case of the more advanced you get in AB testing, the better you are for end development, everybody gets it wrong.

So there’s some fundamental things that people just don’t get regardless of where they are on their journey. And that’s what we’re talking about today.

So let’s first of all look at how AB testing actually delivers value because this is really key to why for an end development should work in the way that it should work, which as well I’m gonna explain to you.

And and a helpful look of it in the financial sense. So This is a view without experimentation. So let’s let’s assume that you’ve got 10 things that you want to do to your website.

So it doesn’t matter, you know, you might have an agency, you might have an in house engineering team, you might have a product team, but Let’s say that the company, the e commerce part of the company has 10 things they want to do to their website.

And they’re gonna build all 10 of those things and push them live.

So this dev cost here, this line here is that is cost of developing all 10 of those things. And some of them will work and some of them won’t work, and most of them will do nothing.

But at this point, no 1 will have any idea of what works or doesn’t work, because they’re just doing things, they’re just building stuff and pushing them like.

But as anybody who’s ever worked in experimentation knows, lots of things don’t work.

Lots of things have the opposite impact that you think they’re gonna have. So some of those things will work, some of those won’t. And you know, you can see here on the right side that there’s a total cost of the development there.

And then, actually, in this case, there is a bit of upside that the the ideas that are working are so beneficial that they’re delivering a little bit of positive and outweighing the negative things.

But the fact is that if you didn’t do the things that don’t work, then you will be in a much better position.

So experimentation looks like this. And rather than build those 10 things, what we’re gonna do is build tests for all of those 10 things.

So that’s the bottom line. So we’re building 10 experiments And only the things that prove value are actually going to be done. So only a test that wins is going to be built properly in the production environment and released.

And then you’re only gonna get the benefit things that win, and you’re not gonna You’re either gonna build nor achieve the damage from the things that don’t work.

So, there’s the obvious benefit of just not doing things that don’t work, only doing the things that don’t work.

But the important thing here is that you’ll notice that the cost to build tests is cheaper than the cost to build in production.

That’s the important thing. It just is cheaper to build and run an experiment in Navy testing platform than it is to properly build a feature in production.

And the 1 example that we get as an agency of this all the time. Is we build tests for our clients, and we might spend 4 hours building a test. And then they will say, this is 1, you should go and push it live.

And then they’ll come back and say, we asked our agency about this, and they’ve quite made 2 days to build it. When you only took 4 hours, what’s going on? And we always get asked, is this agency ripping us off?

Now, sometimes they are. But mostly, They’re not. It is just a much bigger process. Like, there’s a lot more process and stuff that goes with building into production that you don’t need when you’re building an AB test.

And that’s what I’m gonna talk about, but that’s very important. It is much more efficient to build an AB test. But the other very important thing is that you shouldn’t always be building exactly what you’re eventually going to build.

You shouldn’t be testing exactly what you’re gonna build. So the value from experimentation comes from taking an idea and saying, what’s the smallest possible thing that we can test to validate this idea?

The the easiest, cheapest thing we can build and test that will say yes, this seems like it’s something worth pursuing.

So you should always be taking things that are potentially big builds and breaking them down into smaller things anyway.

And again, that’s saves the cost. So a few concrete examples of how that works. You have an idea like should we implement PayPal or Clana.

Now to test that, in theory, you’d have to you’d have to create that capability. You’d have to deploy those things to your website. You’d have to do all the development work around it.

But no, you can take that idea and you can break it down into something very small, what’s called a painted door test. If you’ve never heard of that, painted door test is is that the idea comes from the idea of a fake door.

Now, with PayPal or Clana, what you can do is just put the Paypal or Cliner button in your checkout and when somebody clicks it it says, sorry, this isn’t available right now.

And you get to learn who clicks it who clicks it buys anyway, who clicks it and drops off. So you can make a little case for whether or not you should do it. Second 1, should we allow people to review products on our websites?

Now again, you you know, if you if you just take the idea of let’s allow people to review products, the end goal is you’re gonna have this whole functionality with back end stuff capturing reviews and pushing them back onto the website and management of it and all this kind of functionality that’s a big feature.

So either you go ahead and build that or what I’m talking about, you could say what is the smallest thing that we can build that tests that that this is worth doing.

Now, what you might do is offer some very, very basic functionality basically, just give people the ability to submit a review.

It doesn’t do anything other than play back, but it might be captures it into a Google sheet or something like that.

And you’re just testing the concept, if that was there, would people actually use it? It might be that nobody would use it. And then, you know, there’s not really much point in continuing.

Third 1 should we invest in new product photography. You know, a good example of like somebody said, oh, we should probably hire a photographer and may take all this new photography.

Would cost a fortune. Again, what’s the smallest thing we can do? Okay. We’ll just go in somebody, get somebody to take a few shots on a few key products.

And test whether having more product photographs actually increases conversion rate. So these are just examples, but The key thing really is that to get the to get the benefit of AB testing, development has to be efficient.

You have to from your own process in terms of how you come up with experiments, be able to break things down into smaller things, but then be able to build them efficiently as well.

So that’s the key thing. It’s about building and deploying tests efficiently.

AB testing is about quick testing to validate ideas Otherwise, you’re wasting money on development. You’re you’re gatekeeping the development investment by testing things before it goes into it.

So the whole process has to be efficient, and that’s the key thing to to keep in your mind. So, where did it go wrong? Well, there’s 5 big things that go right here.

The first 1, so in the world where you’ve got a CRO, an in house CRO, or something like that. Running tests, and they managed to get some results from a developer, usually in a dev agency or an in house team or something like that.

What typically happens is that that developer is literally a developer. So they are They will They can code and they can’t necessarily do anything else.

So what happens is the CRO has to completely micromanage the process of development. So that would mean, you know, if I want to build a test and rather than say I’m thinking of testing this, I kind of think it’s gonna look like this.

They’ve got to write an incredibly detailed brief, try and do their own user experience design of some description, and then actually micro manage the process of development.

So every tiny little change is happening backwards and forwards with the developer that doesn’t look quite right.

You need to move it there. That looks weird. You know, you you backwards and forwards. So so the developer is a developer. They can code And yet, they don’t necessarily have any wider skills.

So, in these situations, you literally may as well learn code yourself and do it yourself. It would be quickly like to invest the time in learning to code would be a little bit of an outlay at the beginning.

But then in the long run, you know, you you would do it in the same amount of time that it takes you to micromanage that process.

So you’ve got Yeah. The ultimate problem is you’ve got You have a person who can code, but that is not enough.

You need broad skills. You need somebody who has some initiative around understanding the wider thing that’s going to happen, some design and creative skills and things like that.

The second 1 is thinking that the scales are just counting, so this is an evolution of the same point really.

So most people think, you know, I need I need some developer results and they go out and say, I need somebody who knows HTML, CSS, and JavaScript.

But it’s not as simple as that. So there’s a few key things there. Well, 1 is that Most developers come from a world where they are building things from scratch.

So web development agencies tend to work on the basis of building new websites or building new features in websites and things like that, new pages.

A lot of developers from those worlds are used to building new things, and actually building you know, modifying existing code and trying to get under the skin of what’s being written already and ensuring that when it kind of And if it fails, it can fade fail gracefully is a different type of coding.

Number 2, code can behave slightly differently in AB testing tools, so just because you know code, you also need to know AB testing tools, and a really great example of that is things like single page app frameworks, JS and things like that.

So different AB testing tools work differently with those kind of frameworks. And so it’s not just a case of understanding code. It’s a case of understanding how the code interacts with the AB testing platform and how it works.

And the third part is about is coding is browser centric, not code centric, and that means you know, JavaScript tools particularly modify things dynamically as they hit the browser.

So whereas production coding is about coding into the server environment. So that’s a difference which is a nuance, which can can slide things down and get in the way if the developer doesn’t understand what they’re doing.

Copy and production process So I mentioned this briefly before, but, you know, a web development agency, a product team who are used to building things to feature release following a lot of processes that are not necessary for a b test development.

And so a few examples here. If you’re if you’re releasing something to production, you have to test it against every single possible browser that might it might hit.

So that means browser testing on a lots of different old browsers. A kind of a long tail of browsers as it were.

Now, with an AV test, you don’t need to do that because you can simply exclude those browsers from the test, and you might Let’s say you’re taking out 10:15, 20 percent of the traffic, well, you know, that’s something you can deal with as an a b test, but that’s just 1 example of how there are shortcuts to building and deploying an AB test which don’t necessarily mean that you’re putting any of the experience at greater risk.

Just a lot of efficiencies. And these are little examples, there’s tons of things like this. You’ve also got cross feature testing.

So when you release featureally something, you you know, need to ensure that it is going to work in tandem with any of the releases that happen and there’s a lot of checking of different releases that are happening, might happen in the future and things like that.

Now that is still important with AB testing to some extent.

But not so much and the key thing as well with all of this is that an AB test is very very easy to back out and you just switch it off. Right? When you’re releasing things to to production as features, it’s not so easy.

And so any you know, AB testing is about accepting some risk, and and but the risk is less when you can just switch stuff off. And then the other thing is about the processes that surround approvals and QA.

So, you know, in some businesses, you might try and build an AB test, and it will go through all of the same QA and approval processes, all of which are not needed.

And so it’s going up through layers of bureaucracy to get approval, when again, you might only be testing it for a week, 2 weeks and it might not even you might not even then going to push that exact thing live, not just be learning from it.

So there’s all sorts of things you can do to shortcut the processes that go around production if you know what you’re doing.

So, an interesting thing, so in some of the bigger organisations that I’ve consulted for, 1 of the things that you often find is that You’ve got people running experiments, finding really interesting winning tests, and then trying to get them pushed to production.

And and they really slow down, and they can’t get things done.

And what are the things that often comes out of that, and and 1 of the reasons that people struggle to get time from developers to build AB tests is that a lot of developers just simply don’t like doing it.

So, AB tests are typically HTML, CSS, and JavaScript.

And for a lot of developers, that’s very basic coding. And and a lot of the experiences that are being coded so, you know, by default things are usually quite simple, simple structural changes pages, things like that.

Those are quite simple things. A lot of developers would much much rather be working on complex single page apps, native mobile development, I a augmented reality.

And, you know, these kinds of things. That’s, you know, a lot of developers see their careers being tied to them being involved in those projects.

So you really struggled to get developers to do things because you’re effectively asking them to do things that are not particularly interesting, which slows things down.

And then the last part Sorry, I think I’ve just did some thoughts on my screen. Can you still say my screen? Anybody?

Yes. Yes, Johnny. Yeah. Yeah, okay. So I got a message saying it’s gone off. And the other thing is that Just like I talked about at the beginning, the the the thing that really delivers value is that efficiency in testing the water.

You know, this pictures here for a reason, it’s about, you know, should we do this thing Well, you know, you could you could jump in and do it or you could test something.

You can put your toe in the wall. You could just see if the ice is gonna break, you know, you can do the smallest thing that you can do to test.

And that and the processes that go around it are a skill set that you won’t get by just getting a developer. So those are some of the things that really go wrong.

How do you get it right? Well, And on the 1 level, it’s about the skills. So if you are looking for a developer, you know, a lot of people will just go and find a developer, you know.

And and the and the the wrong thinking is that developer, that code is code. And if you know a code, that’s all you need. But there are particular skill sets that you look at for a developer to have in the world of AB testing.

So they need the technical knowledge obviously. They need to understand code that’s basic. But there are other things like new the knowledge of AB testing tools and how they work and how the Cogases work.

Nuance of coding for experimentation, which is the stuff that I’ve been through. They’ve got to be able to understand how to drive efficiencies and how cut corners safely without putting experience at risk, this sort of stuff.

Operationally, can they manage efficient QA processes? So, you know, you ask you know, and and this might be a developer.

It might be the team around them, but you know, the QA is fundamentally slightly different. You you definitely don’t wanna cook corners that mean you’re gonna break the site because you absolutely can break a web site with an AV test.

But just like that example with browser testing, there are things that you can do to drive efficiencies. Can they fit validation processes?

So that’s that’s testing the water thing. You know, are you gonna get are you gonna get a developer who will help you helps suggest ways to do things like that because a lot of it just comes down to the way it’s built.

Initiative, but this is a really important bit, like, you know, if you ever if you ultimately have a developer who is just sitting there and waiting for you to tell them exactly what to do, then you get into that micromanagement problem.

So it helps if you’re you have a developer who understands the wider processor conversion optimization understands the works of process of digital marketing, what you’re trying to get is involved in the ideation aspect and things like that.

And can take an initiative to do things like the creative. So, you know, in in some cases when you build a test, you need proper user experience design, you need proper visuals for a developer to work from.

But the nature of every testing means it’s not always gonna be relevant or efficient to go and get wireframes done just to kind of move some things lightly or to change something.

So, but if a developer requires that, then you let outlayering in a new process. So if a developer has some experience with design and visual elements, then you’re gonna get a lot more out of it.

And then the other thing is really about the development process. Now, this is what I alluded to right at the beginning in terms of efficiency.

And whether you’re whether you’re a 1 person in you know, 1 person’s CRO in a business or in a product team, it makes no difference. The process should be the same.

You have to derisk development by by treading carefully on the ice and that needs to be built into all of the parts of the processes. So if you if you’ve got product teams then experimentation needs to sit carefully within that process.

The what I often find goes wrong with product teams is what I call cart before the horse. So ideas come in and the product team will go, yeah, let’s build that and I’ll spend 3 months building a big feature.

And then feature release test it. And after everybody spent so long working on it, of course, nobody wants that to fail.

So whether it, you know, a lot of the time they won’t bother testing it because they’ve scared of it not working. Or if they do, they’ll usually end up finding ways to make it look like worked.

Because of course, if you spent 3 months working on something, what happens if it doesn’t work? People will think there’s no value and what happens when there’s next restructure happening and all that kind of stuff.

But the mistake is experimentation is the wrong place. You should be testing to see whether to bother to do the work in the first place rather than testing the thing that you’ve just spent all the time working on.

And there are stages of that like the very first thing like I say. What’s the smallest thing you can test? So you might call that like just a validation test or a proof of concept.

Then you could, you know, do minimum viable product in our gile language which might not be the full feature that you’re gonna release. And again, let you know that that should be the process in any kind of experimentation.

And developments, it’s at the heart of being able to enable that with the efficiencies that go with it. So, you know, what are the key things you need to do?

1, here’s my shameless plug, but you know, if you cannot find the right developers internally, if you cannot, you know, if you cannot get the right developers from your dev agency, there are specialist agencies out there.

We have a pay as you go model for for an end developers for a b testing.

Our developers, all they do is a b testing. So they’re specialists in doing that. I think BW0 might provide similar resources, a lot of the AB testing tools do, but you can use specialist agencies.

Number 2, if you do have developers and you and you want to stick with those developers, you need to train them in CRO and train and get them trained in the wide developments of CRO, what it means, and get them involved in the process.

Number 3, I’ve talked about that problem with unfulfilled developers. Now 1 way to get around that is to rotate developers.

If you have product teams, where developers are, you know, some of them are working on AV tests, some of them are working on bigger features, some of them are working on nice innovation projects, rotate them so that you haven’t got people just bored doing things that they’re not fulfilled by.

And I’ve seen that work very well in quite a couple of in a few different companies I’ve worked with. And last 1 is is have a a separate backlog for dev and experimentation.

So this is quite a specific thing, but it’s quite key. Like, what bit of a story from Sky when I worked there, 1 of the problems that we always had was we found that what came out of experimentation.

We did a huge amount of experimentation, but what came out of it often got stuck in a queue behind other things. And Sky is very political, and you have senior people going, oh, you should do this, here’s this new big project.

And what ended up happening was those bigger things just got kind of churned through at the top of a queue and all the small and medium stuff just sort of ends up stagnating at the bottom.

So 1 of the things we did was to split the queue into multiple things, so you effectively have 3 backlogs of, you know, big projects, small changes and medium things.

And then another 1 actually for supporting the build of experimentation.

So think about the processes that go with that. And that’s it. I think I’ve probably used it quite a bit of time, but I’m free if anyone wants to stay on and ask questions. Thank you for this incredible presentation, Johnny.

And I’m sure that there are a lot of questions we can take right away. So Lemann’s Lemann’s grave has got a question. Kemann’s I’ll be unmuting you. Please do ask your question and any other doubt related to it.

But I’d love to Yeah, Lemmons. I wanna mute it. Go ahead. Did you only only make me know? Okay. Sorry. I was already going. Thanks, Johnny, for the presentation. So far, super insightful and really appreciate it.

Not sure if you can go that deep into experimentation today, but I’d love to hear a bit more from you about running multiple experiments concurrently with the same primary metric measuring the same primary metric and how you usually view the risk of cross interference, especially when you launch them at different times, and you close them at different times.

So usually, we we release almost daily. So how do you how do you view that? What risks do you see?

Yeah. So I think I think the best answer to that is is about is the way booking dot com work. So booking dot com are the sort of poster child for experimentation, the most the most kind of advanced experimentation outfit in the world.

And they famously will run allegedly thousands of experiments concurrently on a single page without worrying about it at all.

So I actually ran a survey on LinkedIn, ages ago, a couple of years ago, actually, asking people what they did, whether they, you know, only ran 1 test on a site, ran multiple tests, but never on the same page or you know, whatever.

And and actually the outcome was interesting. The vast majority of companies that you would class is more advanced. Don’t worry about that at all. They will run concurrent experiments.

And really like from a statistical point of view, there’s no reason why not to do it because if your tool is randomizing things correctly and there are no technical flaws in how it’s working, you are always controlling only for the 1 variable that the test is controlling for.

And even though it might contain people who are being randomized into other experiments, then that’s always identical from the control and the variant.

So there’s no reason statistically or technically why you shouldn’t do it. 1 thing that can happen is if your tests are polluting each other from the perspective of user experience.

So an example of that would be you would have 1 test that’s telling you to click a button and in the other test you’ve removed the button So, you know, in 1 of those crossovers, you’d have somebody seeing it a thing saying click that button and then there was no button there.

So that’s what you have to be careful of and you need some kind of guardrails and way of test of checking that tests aren’t actually interfering with each other from a user experience perspective.

You can also do a form of analysis that looks at the interaction effect between tests which is is not particularly difficult too hard to explain without showing you and but it’s basically a way of looking at whether there has been significant interaction between 2 different tests, which you can do on an ongoing basis.

So there are there are numerous ways around it, but in general, I would say don’t worry about running concurrent tests at providing, you know, tooling is right and they’re not actually interfering with each other from a UX perspective.

Awesome. Thank you very much. Be on a meet. Okay. Sorry. My bad. Thank you so much, Lemans, for I I hope your question got solved.

I have another question from Joe. Joe, I’ll be unmuting you. Please go ahead with your question and take it directly with Jordan. Just a second. Joel, you can go Can you hear me okay?

Yeah. Yeah. Perfect. Hey, Johnny. Yeah. Cheers for your friend. Cheers for your presentation. Well, that’s great. And I found your comments on, like, dialing back to QA process and browser testing kind of really interesting.

And I was just wondering what what browser testing tools you actually use a journey further for a like a b testing, whether it be through VW0 or otherwise.

That’s a great question. I actually don’t know. I’m not directly involved in the development process, but I wouldn’t my developers would be able to tell you.

I know we It’s a combination of we have a a cross browser testing piece of software and but the guys also have a range of devices for for real testing as well.

And the other thing as well is that depending on the client, we will also face with their own QA and testing processes that they have.

So it depends, you know, how we’re working. So it’s not always the same for every client. But yeah, I I you know, I think the thing to reiterate is I’m absolutely not saying that you can be fast in leaps with QA and not bother with QA.

But there are there are absolutely efficiencies and and corners that you can safely cut like the example again.

Makes sense? Cheers. Thank you. Thank you so much, Joe. Teira says thank you. She found the presentation very helpful. We all led the data.

Thank you so much for the comment. Vaz, I’ll be unmuting you next. Please go ahead with the question. As you can Hi. Can you hear me, guys? Perfect. Yeah. Hi, Johnny. Thank you for the presentation. It is very informative.

I don’t cover my technical background. So my question is really where in in a place where we we’ve got quite an old website which we are looking to have a a new website done. We’ve got lots of ideas for this new website.

But my sort of ready is a bit like the on your presentation, you showed different ideas. So if you put all the ideas live from day dot when the website goes live. We won’t know which ideas worked and which ideas hasn’t worked.

So is it a case where we do really sort of simple website without, you know, the 10 or 20 ideas that we have had been experiment with those ideas rather than doing them from the start.

Yeah. I mean, I don’t know how far you are down the process, but I would say there’s there’s generally very little reason to completely redesign a website.

I would always advise not to do that because if you really think about it, you you know, like, it it when when people say they need a new website, if you if you really kind of question why there’ll be some There’ll be several things that they’ve, you know, come up with like, you know, 1 of them might be where we need to replatform.

Well, if you need to be platform, you don’t necessarily need to redesign the website. You can, you know, you can pretty much transition to the same website, their same website.

But then other things that will be like, well, you know, with the checkout, which everything is optimal. Okay. Well, exactly what don’t you think is optimal. It’s in 3 sections where I was thinking of it might be in 2.

So what you what’s happening here is you’re coming up with test hypotheses. You’re coming up with, you know, individual ideas and you can test these all these things without redesigning the website.

A grant you know, clearly understand that people do redesign websites happens all the time. So if you if you are absolutely have to do it and you are doing it and it’s happening, then, yeah, you’re right.

I mean, 1 of the things that you can do is, you know, if you can to go through that process anyway, if you’ve if you’ve already redesigned the web and it’s in and it’s in progress.

You can break down the key the key differences between the new and the old you know, the theories that you’ve got about how the website should change and still test them on the existing site because you might be that you’re wrong, and you might wanna back most things out of the design process or or if if nothing else, you might just learn something about how users behave, which might help you understand how better to execute that particular thing in the redesign.

So really a redesign should should have testing that leads up to the actual launch that helps inform the design. And then exactly as you say, you might then have a whole series of things that you wanna test immediately after.

And I’ve honestly, like, can’t tell you the number of times in my career that I’ve worked with clients who’ve done big redesigns and they’ve just it’s been disastrous and that, you know, like, they’ve instantly seen a big drop off in, like, all sorts of metrics.

And and it and it’s a bit It’s just a gamble really, like But yeah, if you if you are doing it, I would say the number 1 thing you can do is kinda go through what that redesign is and write down all the things that all the hypotheses that are baked into it like, you know, we’re changing this because this, we’re changing this because this and and find a way of testing those things either before or after or whenever you do it.

But yeah, I think just to go back to us at the beginning completely depends where you are in the process and sometimes, you know, if people are far along then it’s just a case of like you say, testing things afterwards but It’s an interesting area.

As I hope that answers your question. And just to add on to Johnny’s point, it really depends upon where you are in the journey to, you know, kind of understand your needs and requirements.

I think that would be the okay. I see another question. Yeah. Yeah. So we will be sharing the recording and the presentation. In the and obviously, a bit later. I think that would be it, Johnny.

Thank you so much so much for the presentation. It was super insightful. I’m sure a lot of these learnings are going to help a lot of teams be technical, be sound about their product development and testing development journey.

If you are if you have any questions, do redirect them to us at marketing at beta duo, and we’d be more than happy to, you know, get you answers from Johnny.

Otherwise, Johnny is a big time influencer. You can reach out to him on LinkedIn and his socials to, you know, kind of get your clarifications. We we are obviously, you know, bridging up more such upcoming sessions.

Do stay connected to VW webinars for, you know, any more doubts regarding experimentation, regarding for development, personalization, and everything about, you know, developing experiences.

So thank you everyone, and thank you, Johnny, once again, for the session. Thank you very much. Have a good day. Bye.