Personalization at the Speed of Intelligence

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Hello, and welcome to this panel discussion at Convex twenty twenty five. I am super excited to be here today joined by my cohost, Sandra and Erica.

Today, we’re gonna be talking about meaningful experimentation. And to kick things off, why don’t we each just introduce ourselves and tell everyone who’s in the room and what we’re passionate about. Erica, do you wanna kick us off?

Yeah. Absolutely. So hello, everybody. My name is Erica Sense. I am the manager of events marketing at WP Engine. I’ve been at the org for probably I guess, I actually just hit my four year anniversary, which is crazy. And I’ve thank you.

And I’ve had the privilege of working mostly in website strategy and conversion and optimization for the last couple of years and only recently took on this events purview. And really what’s cool for me is it’s just another way to experiment and try new things. So it’s it’s getting to flex those experimentation muscles in a completely different way in person and online, which is a really approach.

I love that. Sandra?

I could go next. So hi. I’m Sandra. I’ve been working at World Advisor for about over five years now. And I joined the organization in twenty twenty.

And I’ve been there since the start since they were just a start up. We initially started off in the American market. Fast forward to today, we’ve sort of had a we’ve grown into a global, brand presence across UK, Canada, Australia, India, and Germany.

So it’s just been a wild ride for me. It’s a bit about myself. I started out in the growth space, so producing and helping build and scale our content strategies, working on optimization. And now I’ve sort of moved on into working in product where I sort of focus on our proprietary products, our user journeys, and also testing optimization for our landing pages. One of the fun projects that I’ve sort of been working on since early this year is leading a UX team. So that has been honestly one of the most exciting chapters so far for me.

I love this.

And what I love is that everyone’s coming in from different marketing parts. Right? So I’m Talia Wolf. I am the CEO and founder of Get Uplift.

We are a conversion optimization agency, and we focus on driving meaningful conversions through psychology, through emotional targeting. And we help brands in b to b and b to c service based companies understand who their customers are on a more deeper level and then create better experiences for them and run, you know, experiments. And what stood out from both your introductions is that it’s very similar to mine in that we all came in from other walks of life in marketing. Like, did social media marketing.

I did paid media before, content strategy. We’re all coming in from multiple types of marketing.

And I think that is the key when it comes to experimentation because as opposed to maybe PPC or SEO or other types of marketing.

When you’re doing CRO and experimentation, you have to know a lot about a lot. And as you said, Erica, it’s it’s not just all the website that needs optimizing. It’s every part of the customer journey. And sometimes it’s for events.

Sometimes it’s on your website. Sometimes it’s for email marketing. So I guess I’m gonna kick things off and maybe talk a little bit about something I like to call the hamster wheel of optimization. And it’s where most marketers are at when they’re trying to drive results and they’re stuck.

So you’re throwing best practices at it. You’re copying competitors. You’re doing you’re trying to reverse engineer whatever, you know, some guru said on LinkedIn. And more and more, you’re just throwing tools and software at the problem.

Maybe you’re getting an AB testing platform, maybe personalization, maybe you’re trying a new landing page builder, whatever it is. You’re kind of throwing stuff at it, but nothing’s getting you the results that you want. And that is super frustrating with CRO. So I wanted to ask you both about before even running tests, how do your specific teams approach experimentation?

How does it impact, you know, the rest of your team? Do you actually work with other teams other than marketing? And do you feel that where you work in experimentation, is it different in your industry, in your category? Is it different than like the conventional CRO that we see on a website?

Sanjana, do you wanna take it?

I’m happy to kick it up. So just with regards to running tests, before we run any tests, our intention is to understand alignment and make sure that we are focusing on user intent. So when we run experiments, it’s not for the sake of changing a button color, for example. We want to lead with data.

What exactly are we seeing on our landing pages? What are our users trying to tell us? Again, with Forbes Advisor, we’re working across various product categories from insurance, credit cards, to all the way to, like, mattresses. So the kind of users that we are dealing with have such a different approach when it comes to them, clicking on our pages, clicking on our products.

So what we do is, one, try to understand the intent. It could be commercial, it could be informational, it could be transactional. Besides that, also mapping the stage where the user is in terms of the funnel. So at the top of funnel, middle of the funnel, bottom of the funnel.

And to be able to kick off this testing, primary agenda is to first gather data to be able to justify testing. So again, going back to GA4, pulling queries from BigQuery to be able to understand where we are seeing friction in terms of understanding where the conversion rate is. Are we seeing bounce rate for go high for a particular landing page versus another product category, so on and so forth. Besides that, again, you want to sort of marry that with qualitative data.

So you have clarity kick in, you’re looking at heat maps, you’re looking at clicks.

One thing I do love and enjoy is looking at those user videos. It really, truly tells you how the users are interacting with your page. You expect it to be this really proper way of them to, in terms of interaction, you think that they’ve sort of dedicated their time and their life to looking and reading through everything that you’ve put on this landing page, but that’s definitely, of course, not gonna be the case. And ultimately, combination of that quantitative and qualitative data sort of helps us understand what we should be focusing on. And as people in the experimentation field, I think the meat of our job job is to define the problem, define what should what should we be focusing on to focus on those high impact areas.

And just with regards to who is sort of involved with in this experimentation process, aside from the marketing function, we’ve got designers, we’ve got engineering teams. Again, you wanna closely work with editor in our case, editorial, SEO, and SEM teams to be close to where the users are, to understand that in time, to understand how they’re creating content, how they’re making sure that they are simplifying a lot of complex processes for the users.

And I think that’s how we sort of approach our testing process in a more collaborative sense rather than having it siloed. So ultimately, each sort of function essentially is a loop where we got insights from each function, whether it’s your editorial analytics or your SEO, SEM, and that feed feeds into each other.

And we’re able to ensure that we we make decisions that are guided in our data. I think that was a very long answer. I’ll let Erica jump in.

Yeah. I would say so I feel very fortunate to work for an organization that has already has always really prioritized experimentation. Our leadership believes in it, which is hugely important when you’re trying to achieve something and you, you know, you wanna make a pro a program of success. So when we are working on experiments, whether it’s on the website or it’s an event or it’s our email marketing team or it’s our one of our paid teams, you know, it’s a really collaborative effort.

We’ve we work very closely with product teams. You know, we work closely with sales to understand what they’re hearing and what our customers are looking for and prospects are looking for. But for me, when I’m thinking about an experiment, it really it it goes past just like thinking about the experiment itself. It’s not just what, you know, our competitors are doing or what other websites are saying or what what’s working on LinkedIn.

I really do think it’s about understanding what your customer is actually looking for and understanding who you’re actually targeting.

We have an amazing CMO who has really like, performance matters. Performance is incredibly important, but she’s she’s been a great reminder that you have to keep the customer at the forefront the entire time. I think a lot of times we’re like, oh, it’s just another click. Right?

Like, we just wanna make more money. We just wanna drive this goal, and we get so drawn into those KPIs that we’re trying to drive that we forget about the person behind it that’s actually taking that click or is giving us that money for a service. So understanding who they are and, Sandrana, you mentioned, like, where they are in the funnel and, you know, what they’re doing and where they’re doing it. And that’s so important because how do you know what to drive or how do you know what to change or how do you know, you know, what needs to be improved if you don’t understand, like, what their desires are, what problems that they have that your product needs to solve.

So it’s really getting to the heart. And we have we have an amazing segment marketing team who has an understanding of each segment within our audience where you can go in and say, okay, like, let’s have this conversation. Let’s talk about this and let’s figure out what their desires are, what they’re looking for in this product in order to then build an experience, whether it’s in person or online, that will drive the results that we’re looking for.

I really love this because I think as marketers for years now, we’re all being told to be data driven. Like, you have to know all the data, and we are collecting data like Pokemons.

And it is insane. But what happens is we collect all this behavioral data, so we go into g a four or whatever analytics system we’re using. And we have all this behavioral data. We have age, geographical location, we have gender, we have browsers, devices.

We know all this stuff. And that tells us the what. It tells us the segment. And in the in in our analytics, can also see where the problem is.

So it’s a product page. It’s a sales page. It’s the home page.

We can see where the problem is. We can see who is not converting. But the biggest question most of us have are, you know, okay, what changes do I need to actually make to increase conversions? I mean, it’s great that it’s my landing page, but how do I know what exact changes do I need to make to my page in order for it to convert?

And that’s where qualitative data lies. And so many of us completely forget it because it’s much easier to look at a heat map. It’s much easier to look at GA four. It’s a lot easier to maybe look at some recordings.

But doing the actual work to uncover what our prospects are feeling, what their pains are, their challenges are, how they feel about themselves, how they want to feel after finding a solution. That’s that’s the big work that we need to do because at the end of the day, there are hundreds of products out there just like everyone else. So we’re all competing with everyone, and we all sound the same, especially now with AI. Right?

Like, you can you can just, like, push out anything you want all day long, and it ends up with, this AI slop. Everyone looks the same, sounds the same, and we’re making it harder on prospects to figure it out. So qualitative data, I believe, is, like, where the money is. It’s where you can really connect with your customers.

And for us, it’s been in things like doing a Reddit analysis. So literally looking at conversations and threads that people are having in Reddit about whatever it is in that industry and figuring out the sentiment. Or looking at Quora, LinkedIn. Also running surveys, customer interviews, and and stuff like that.

So I would love to hear from you. Is there any, I don’t know, an insight you’ve ever collected from qualitative data that you wouldn’t have found in a dashboard and, you know, that it impacted an AB test you ran?

Honestly, I think if anything, a lot of the qualitative data confirms those sneaking suspicions that you, like, you have in your gut, but you can’t back it up with data.

And sometimes that’s okay because, you know, it is what it is. And, like, you sometimes you just have to do it because, you know, your gut is telling you that that’s the way to go, and there’s no way to back that up. Like, you you’re reading through Reddit and you’re reading through LinkedIn. I have this feeling that people are looking for this thing and they’re not getting it from us.

So for for us, one of the tests that we ran probably almost a year ago was really starting to change our messaging completely on our website. We were really missing the mark from a value based perspective. And, like, a lot of times you would go on the site and you’d be like, okay. Like, I get what their product is, and I get that they’re selling managed hosting for WordPress sites.

But I don’t know why I fit into this. And the more we read, you know, what people were doing and, you know, the challenges that they were having on all of those sites, we, you know, we realized like, hey, the website’s not even answering their questions. The website’s, you know, it’s performing well and all of these tests where we’re moving things around or adding things to the plans page makes a lot of sense, but it’s not it’s not getting to the emotion of it. It’s not getting to, like, what they’re physically looking for.

And the data, like, the data doesn’t show that. Right? The data shows that we’re still making money, that people are still clicking on the things.

But at the end the day, like, made the changes and suddenly we’re seeing a huge upshoot in engagement. And it was, you know, it was all based on like, hey, we have this ink like this little, like, inkling feeling that it’s happening. And but, like, none of the research that we’re do like, that we are doing, of course, our UXR team sit in there like, guys, we have this.

Look at all these look at all these interviews that we’re doing.

Or, like, they’re saying this out loud and you’re just not answering the question. So it’s really it’s like, you just sometimes at the end of the day, you have to listen to the person and not the data, which he and I know it, like, pained all of us to say, like, that hurts my soul to say that. But, yeah, sometimes you just have to trust your gut. And hopefully that answers your question. I feel like I went on a little bit of tangent there.

No. I agree. And I and I will say that I think a lot of people struggle with qualitative data. Right?

Because it feels very abstract and kind of gut. Like, let’s go in our gut. But as you said, your UX team has probably interviewed people. They probably surveyed people.

That’s kind of the silos that sometimes happen. So actually pairing it with, okay, read all this stuff on Reddit, then I’m surveying my customers, and I’m also maybe putting a visitor survey on the website. Maybe I’m also interviewing five or ten customers. You can actually start to see a pattern very quickly.

And it’s really cool that that was the thing that you put up in focus. So I love that. Sunjana?

I think I agree with you, Tania. When you sound like numbers don’t tell you the entire picture, they don’t tell you the whole story. And like I mentioned earlier on, I was responsible for leading a consumer research team in our department, whereby it was me and a bunch of stakeholders from the UX team. And our motivation was to be able to learn and anticipate a user behavior. So not just keep this as something for later to diagnose a problem when the friction happens, but to constantly evolve and learn from the kind of behavior that we’re witnessing.

So again, we wanna dive into We were able to do a whole lot of quantitative surveys, ads, and unmoderated and moderated studies as well. I think a lot of really, really good data points came out of doing the moderated study. One that comes to mind is a travel insurance moderated study that we ran for people over the age of fifty years old. And an interesting data point that came out of that was they were unable to understand the terminology when it came to travel insurance or products.

So they basically needed a bit bit more explainer or a bit more definition in terms of, like, what they were clicking at, and they were unsure as to what kind of data they were meant to be entering.

And I think this is something, again, you could look go to Clarity, look at the videos, but you couldn’t really put a finger to up finger as to why this was happening. You could see this was a problem, but you could not understand why there was this hesitation. That’s when sort of like qualitative data, wherein conducting these moderated studies had these incredible values. So ultimately, I do understand how important it is when data will tell you where to look. You do need the qualitative study to exactly sort of ground your hypothesis in human behavior and human motivation.

Yeah. That’s super cool. I love when there’s so much data to go into.

And I mean, so Forbes Advisor serves readers on very, like, high stakes decisions. How have you experimented with, like, content or user journeys strategically to help these kind of decisions? What are other kind of experiments you feel that are more strategic that you’ve ran than, like you mentioned, we don’t do the button testing? I mean, button testing is fine.

Don’t let’s not get into that. Alright. No. Sorry.

But generally, like, what are some strategic tests you’ve been running that are different? Because obviously with high stakes decisions, it’s a big thing.

Just to touch very briefly on the button testing, I think they’re completely fine. I think they just have to be rooted in reasoning. That’s that’s that’s the basis you want to sort of start off with.

Yeah. And I just I’m gonna hop in. Sorry. I’m just gonna hop in. I was gonna say, like, we ran a button test for one of our clients last month.

And I’ve been the whole I’ve been for years like, stop button testing, stop changing elements in the page for the sake of changing them. But we run a we ran a button test because we realized that was there was a a very big disconnect between what we the the action we were asking people to take and the result they were going to get. So when there’s a problem you’re trying to solve and there’s a clear hypothesis behind it, button testing is great. It’s just the, like, none of us most of us aren’t Amazon or Booking dot com that can, you know, change the the shade of orange and see if we get any results. That that’s that’s my take on it. Sorry. But please go ahead.

No. I I see I mean, I definitely see the value in having to, again, test with buttons, test with a lot of smaller elements on your pages. As a Forbes Advisor, we’re sort of constantly looking at iterating our high impact landing pages. So again, this is across a bunch of product categories that we’re looking at from credit cards, home improvement to insurance.

So we wanna be able to again focus on these high impact areas. Just like mentioned, the button test, we’re constantly sort of iterating to understand how smaller element can also sort of weigh into getting that lift from the users. I think a test that I think everyone in the experimentation space is sort of like honestly bored of, but I think is incredibly helpful if the hero panel testing. And I’ve done like a billion of those with different itinerations.

So having a CTA button, not having a CTA button, sort of focusing on the MVP or just synthesizing what the page sort of talks about and things of that nature. Something that’s worked incredibly well for us after sort of testing these out was, one, we wanted to be able to understand what works well, and then secondly, can we scale that? So in terms of our hero banner testing, what we found was we got the this variation that worked out was for us was having a CTA button along with focusing on the MVP, so the product itself. So directly speaking to the users about the product because it was a a transactional page, people were really looking to understand the benefits when it came with regards to the product.

Another test that comes to mind when we were thinking about optimizing our landing pages is adding a secondary CTA button, which is incredible. How about adding twos if they want?

So this, again, is different from, like, the transactional intent. This was for a page on, I think, the health category whereby users were more interested in exploring. So we were able to cater to both high intent and low intent users, and I think we saw tremendous success there as well. Apart from just the landing page testing, we’re also sort of focusing on user flows.

And with regards to user flows, we are looking at categories like home improvement, whereby we have products like solar, windows. And here, I think something that worked incredibly well for us is having tested, like, social proofing, testimonials, and the CTA wording. So in terms of CTA languages, we’re still sort of struggling with making sure that we’re able to sort of meet that intent with every single user because, again, every user that enters the journey will have different expectations out of that. Something that’s worked well has been social proof and testimonials.

Again, that speaks to how context is super relevant for the users because and ultimately, are focused on their end product, and they want a bit more trust and clarity in terms of, like, opting into these user flows. So the way I would sort of look at experimenting, whether it’s with regards to landing pages or with our user flows, is ultimately making sure that our end users are confident and informed. And ultimately, they’re making choices that they believe in. And every test that we sort of work on designing is not just to make the decision process faster, but also to make it easier Yeah.

For the end user.

What I’m hearing and really resonate with is two things. One, you used the word struggling, but the fact that everyone but everyone is struggling in Siro. And I think when you see big brand names, you’re like, well, they’ve got it made.

Everyone’s struggling. Like, you work with huge brands, everyone’s struggling because we don’t know. The other thing that I’m hearing is that the most important thing is understanding user intent, you get that. You’re constantly looking at what drove this person to the website.

What where are they in their stage of awareness? Are they aware? Are they solution aware? Are they product aware?

Are they most aware? In order to give them the best offer that you can. Sometimes that’s an article. Sometimes that’s a product page, sometimes it’s, I don’t know, shopping cart.

But that is super important. And I think it’s it’s one of the most important things we can do as CROs, as marketers, is actually stepping back from, like, taking action and doing the research.

When I think that that’s always kind of hard.

So I guess my next question is for you is for you, Erica. One of the things that I talk a lot about, is emotional targeting. So understanding the emotions that drive decision making. And I talk a lot about it in the frame of b to b.

The biggest question that I get is, well, of course, emotion works in b to c. If someone’s buying a watch or a t shirt, it’s emotional. But in b to b, no. That’s completely logical, and we need to hit them with features and pricing and our technology. And don’t forget to say powered by AI and the number one platform for.

And I’ve been waiting to ask you this question because you specifically, a, mentioned emotion, and b, catered to developers mostly. Like, part of your ICP are developers, which means kind of like a b to b approach.

So that’s I would love to hear from you if there’s anything different that you think that you do that you’re that the wider team does when it comes to experimentation, whether offline or online, when you’re catering to developers in terms of understanding their intent and and how you experiment and serve them on websites?

Yeah. So the funny thing about emotions and when it comes to b two b especially is it’s not as personal. Right? It’s not like, oh, like, I had this thing that that did a thing and, you know, now I’m all torn up about it.

But for us, when we’re thinking about a website, because that’s what our business is, is hosting websites and providing tools to improve efficiency and to, you know, work smarter, not harder is a website is a really for a lot of businesses, it is it is business critical. Right? It is your livelihood. Whether you’re a small business or you’re, you know, mid market enterprise business, if you don’t have a website, you might not be able to sell your thing.

You might not be able to talk about this thing that you are doing, which ultimately impacts your team. And, you know, if your website’s not working, you’re not able to sell the thing. You’re not able to make money, and then you’re not able to, you know, have the people that are working for you. So I I I really like, I I don’t think b two b is emotionless at all, and I think that’s probably what a little bit of what we’re seeing right now with the world of AI is everybody’s turning to robots and forgetting about people when in reality, like, we need to think about the people.

And so it it’s really like again, I feel like I’m a little bit of, like, a squeaky wheel here is, like, it’s about knowing our audience. So with a developer, for instance, they’re going to most likely be a little more logical in in what they’re thinking, and they’re able to you know, they know the jargon. They they can they can read the room.

They get when we’re when we start to get a little more technical. And so we can talk about those things that, hey, like, remember that time you had to stay up and miss your kid’s birthday party? Or one of the stories where I was I was recently reading was like, a a gentleman was not stories, case studies. A gentleman, like, missed their like, a a vacation because they were up because their site went down on another host. And that is emotional.

Yep.

And so understanding those pain points to tie into the, like, the value of what we’re offering That hasn’t always seemed like something you can do to do in b two b, but now it’s so important as people think you’re just a robot or you’re just Yeah. Doing, like, sneaking something to say something. Like, it yeah. Like, people care about that kind of thing.

And if you can hit to the, hey. Like, we got it. Don’t worry about it. Right?

Like, can go hang out with your your family and go on vacation and do these things and your boss isn’t gonna call you screaming in the middle of the night because your site is done and you’re not able to make money.

You know, it it helps us figure out how to talk to them and what to say in a more effective way. So when we are running experiments or we’re hosting events, you know, we know what they’re looking for out of us and out of a partner partner so that and and they’re looking for us to say that. Yeah. Right?

Like, just say it the same way. We’ve seen your struggle. We’ve been there. I mean, I the number of time, like, I freelanced a million years ago and built websites, and I had people who called me in the middle of the night when I was on, like, budget hosts.

It was like, my site’s down, and I you know, it’s almost Black Friday, Cyber Monday, and this is how I’m feeding my family. You know, that that hits to the emotion of it and the root of it and knowing that that’s what they care about Yeah. To get ahead of the game. So how do we run experiments, run tests that hit that home and let them know we’re listening to them and hearing them?

Yeah. Absolutely. And I can tell you that last year we ran for one of our clients, we ran some user tests and we watched them. We asked them to basically look at our client’s website and five other competitors. They didn’t know which one was our client.

And we watched how they basically browsed through a b to b product. And what we found was really interesting. It was, okay, they go to the homepage, then they look at the pricing and they’re like, okay, check, check. It’s in our, you know, it’s in our pricing range.

Then they quickly go into, features. So like, okay. Yes. Yes. Yes. Yes. Yes. Then they go into integrations.

Okay. Great. And then they stare.

And they’re like, I’ve done the checklist. And now everyone’s the same. I need to make a decision. And what we found is that that decision, that critical decision in B2B is when you make it about that.

So if you were able to show them that, like, you’re gonna give them their time back or you’re finally gonna know if Patty from accounting is actually working or not, those kind of things is what actually sells. Because and and that’s why, by the way, you know, proof Sanjana, you mentioned this before is, like, social proof and trust and all this stuff is what really motivates people because at the end of the day, everyone’s the same. And even if you do have one feature, you’re you’re powered by AI and no one else says, yeah, tomorrow someone’s gonna come along and do it. So the that value prop of, okay.

Yes.

I see that. Oh, they’re gonna lessen this pain for me.

That I think is, you know, is the key. So I agree.

Alright. So let’s talk a bit about the elephant in the room, and that is not AI. We’ll get to that in a minute.

Please. The elephant in the room is buy in.

This is a question that I get from many marketers.

They usually say something like, okay. I’m bought in. I understand I need to be more strategic about CRO. I shouldn’t just button test or, you know, do the the best practice stuff and move on.

But how do I bring my team along for this? How do I get the buy in from my executive team, from my colleagues to actually do different things? Sometimes like the different thing is even like, hey, I’m just gonna go over here and comb through some conversations on Reddit, or I’m gonna run a survey or I don’t know. I’m gonna do all sorts of different things in experimentation.

You both have worked with many, many people on different teams, and I would love to hear from you about how you get buy in. How do you you know, what is that what what do you do when you need to, like, okay. I’ve got this. I have an idea, but now I have to convince everyone else in the room.

Sanjana? Yeah.

I I could go for it. I think one thing that I couldn’t recommend enough is hosting workshops. Get people in a room. For me, that’s a remote room, so it’s on a Google Meet, unfortunately.

But what matters to me is that everybody gets to share their opinion. Once everybody shares their opinion about a certain process, I think everybody feels attached to the product, everybody feels attached to the project that we are working on. So basically, making sure that we are able to involve our stakeholders only into the process, make them invest in, make them feel like they’re part of the story. So this also requires groundwork on your end. So for me, it requires that I can Again, my go to is a good old mirror board to make sure that I lay the groundwork. I have a simple framework whereby I’m able to sort of map user intent and potential impact in terms of, like, the test that we want to run.

And just with regards to getting buy in from stakeholders, one, it’s gonna be an uphill battle, and I think that’s the god honest truth. And I don’t think you can change that. I think coming to terms with that is a part of the process.

You can have excellent, excellent ideas, but that does not translate well until unless you’re able to tell that story to your stakeholders. And for me, the way I sort of went about doing that was making sure that I am able to, again, not just host these workshops, but have follow ups with stakeholders, get an understanding of why they have trouble with sort of proceeding with any of these requests, any of these tests. So I think an example that I could think of is I was looking to test a pricing bundle earlier into the journey rather than midflow. This was a high ticketed item whereby the pricing bundle was, like, barely midflow.

And what happened was we had a lot of users kick in, and the minute they got onto the pricing option, that’s where you could see everyone jumping out. Because again, this is a high ticket item. People want to know how much this cost upfront. They want to know that there is immense value in terms of what they’re purchasing.

So one way I went about sort of collecting data for this was I did like a competitor analysis to understand how competitors are dealing with pricing options. Is it in the mid flow? Is it are they prime users early? Is this the first thing that you see on their landing page?

And that was the case for most of the competitors out there and especially the top competitors in that particular industry. So, again, going in with the going in with that data, apart from that, just understanding why they’re sort of skeptical in terms of wanting to not move move to with sort of testing out the pricing option early into the process. So one of the reasons that I I got in terms of pushback was that they’re still looking for a lot more peep users to sort of kick into the journey, which, again, in theory sounds great. We yes.

You want high traffic. You want more user to come to your use to your to your product, but you sort of lose those users mid flow, so that honestly doesn’t make any sense. You want those high intent users to go through the journey so that you are able to sort of lift that completion rate when it comes to that particular product.

So particularly my advice to other stakeholders out there or other people in the experimentation space out there is to approach this with empathy. Understand this is going to be an uphill battle, and sometimes you will have to meet them at the middle ground. And that is something that you sort of just have to work with.

Nicely said, Erica.

Yeah. So I will say one thing is that I do think it’s a lot easier nowadays to get that buy in. I think experimentation like AI has become a bit of a buzzword. And, you know, I find a lot more executives wanna talk about it. They wanna be part of that conversation.

They want to be able to talk about their team doing it when they’re at a conference or they’re talking with some of their fellow executives from different organizations. So I do think it’s gotten a lot easier. Like, five years ago, I feel like it was a constant battle with like, hey, just let me try this thing. Let me show you how this works.

I so for me, aside from it just being a little bit easier nowadays is really like starting small. Right? If it’s a button test, that’s totally fine. There is a time and a place for them where maybe that’s all your organization will allow you to do.

Great. Fantastic. Just start somewhere. Start small and start to bring them data of why you’re doing it.

Right? So Shay, you saw this thing that it looks like people are dropping off at this phase in the journey, whether it’s in the the purchasing funnel, it’s somewhere else. I have this thing that we wanna try. We’re gonna split test it.

You know, we’re we’re very mindful. You know, we walk we’re watching the test to make sure nothing goes sideways. And if something does go sideways, like, we’re gonna be on it so you don’t have to worry about it. So ease those concerns.

And then also, honestly, show them how the sausage is made. Bring them in really early even so I think that the the best place to do this is with your skeptics, is bring in the naysayers, listen to them, include them in the decision making process, allow one of their ideas. Maybe you do an ABC test. Right?

Allow one of their ideas to get put in to show them how it’s made because you’re just gonna make them into a believer. You’re gonna make them into a fan who gets a little ex like, there’s that little high of when your test wins Yeah. Where you just wanna, like, bring them in for that and show them what’s possible so you can keep moving. And you’re gonna have to do that over and over again.

Right? You’re gonna start like, for us, you know, our web team believes in it. And then originally, years ago, it was, okay, my immediate team believes in it. Now I have to go get this other team that we wanna work with.

So maybe it’s paid media for instance. Right? They’re rolling tests on their ads. We have to work together so we’re not, like, who does test want or, like, actually cause the lift?

So you have to start working with them. And you’re gonna have to go through this process all over again of, like, showing them how it’s done, bringing them in early, involving them. And then you win that team, then you have to go to the product team and say, hey, like, we would like, this thing won on our side. Why don’t you so for us, our website marketing website is a separate team from our user portals.

And there’s a lot of similarities between it, but it’s like, hey, we did this thing.

Now we want you guys to try it because it had a lot of success for us and you have to bring them around again in that entire process. So sometimes it feels like you’re just like saying the same thing a million times just and, like, constantly getting to have to get buy in and maybe getting beaten down a little bit on the process and, like, going back around. But, ultimately, at the end of it, like, people are so excited to be a part of it and so excited to be involved and to have a voice and to see whether or not it wins, loses. Like, there’s an excitement in it that, you know, you just wanna see the next one and you wanna see it be successful.

So it’s it’s a hard process. It’s a long process. It never ends. But there’s a lot of good at the end of the tunnel, and it just takes a while to get there when you wanna try and get that buy in.

Yeah. Absolutely. I agree. And, I think once, as you said, you bring in the people and they’ve bought into it a little bit, Every other, you know, test that you run, they’re gonna be rooting for you a little bit more, and they understand what you go through. So, yeah, I think I’d I yeah. It’s an uphill battle, for sure.

And it’s never ending.

But it’s that time for us to talk about the most intriguing thing, I guess. Looking forward, you know, it’s I can’t believe I’m saying this, but it’s November, which is crazy.

Looking forward into twenty twenty six. So people are starting to build their roadmaps and their plans and their strategies, and it’s that, you know, time of the year for planning.

So, yeah, maybe to close out, why don’t you tell everyone what are you focusing on in this coming year?

Are you incorporating AI and personalization into your plans for next year? If you are, how?

Yeah. And what do you think teams should look forward to? How should they start planning for stuff?

I could start off. So with regards to AI, I think that’s the elephant in the room. Everybody is obsessed with it. Everybody is gagging about it. So I’ve seen merit in AI.

It’s just with regards to the approach you wanna go about when you are using AI. For me and my team, the benefit that we did see the most when it came to AI was largely when we were researching and synthesizing our research.

Agreed.

So again, we’ve had a couple of hallucinations there as well. So I just wanna be cautious when I say this. So still wetting the information that you get from AI when you are whether you’re synthesizing information or you’re are looking to research.

Apart from that, something that’s really helped, again, this goes back to the consumer research team that we were working on. And when we’re testing out design prototypes, we’re tools like Lovable, which we’re using other AI tools that help us build these prototypes. Again, this saves us a whole lot of time in running these unmoderated studies, for example. So I think that’s where I see the benefit.

So if you want to test out a feature and you want to speak to the user, you want to see how that they interact, I think, again, it’s a clunky piece of our software. It’s not the best. It’s it’s a good place to start unless, like, you’re waiting I mean, unless you have the resources and the time in the world to be able to, like, sit there, build something, go to the users, test it out again. Most of us do not.

So I think it’s a really good middle ground to use this particular tool out there.

With regards to personalization, I think something that’s worked incredibly well, especially when it comes to Forbes Advisor, has been dynamic titles in the headline. So when it comes to geographies, I think that’s something that does incredibly well. So if you’re from a particular state and you’re accessing one of our articles, being able to have that localized content piece of content really helps.

Besides that, we’ve also sort of tested personalization when it comes to interstitial. So this is with regards to our personal finance category, whereby users are not going to be sort of making that decision.

I mean, there’s going to be a whole lot more thought process when it comes to a category like personal finance and in terms of adding that interstitial and personalizing it with the products that the end consumer opts for, I think that has been beneficial. We’ve seen a statistically significant uplift there, and that is something we we can’t really scale that across our other product categories, but I think that’s deeply valuable to the personal finance product vertical that we’re dealing with. As to what can sort of users and stakeholders look forward to in terms of next years? I can’t sort of press enough on the fact that please do external showcasing.

Please get up by it. You need to be able to have not just senior management, but people from, like, different teams. Yeah. Love and value what you’re doing.

And the way you can go about doing so is telling them the incremental revenue, telling them how these tests were run, telling them how sorry.

What the user behavior was with regards to running these tests. I’m going to be with regards to AI, I’m going to be continue being skeptical about it. I love it when people are talking about it on LinkedIn, but in terms of practice, I’m I’m gonna be a bit skeptical. So I think that that’s that are my thoughts so far.

Yeah. That sums it up for sure. Great. Erica?

Yeah. So I have a really cool opportunity where I’m planning for something that is not you know, it needs a little more we need a lot more lead time on what we’re doing now in the event space versus when I was in experimentation because you have to give people the money and you have to sign the contracts and you have to have the team ready to go.

So planning for me but knowing that, what I’m actually state taking a step back, and I’ve been working on my strategy deck for next year. The biggest thing for me has been, okay, how do we do this year? Right? It’s pulling in that data.

It’s using AI. So, Donna, like you said, like, being a little skeptical because it does make some things up, when I’m I’m reading some results, but saying, okay, we did all of these events this year in person and online. How’d they do? Right?

Like, what are your takeaways for each of these segments, for each of these objectives? Did we achieve that? You know, is there something that I’m missing that I’m not even thinking about? And then taking all of the data in, we we very similarly to how we how we do it in web is we have an understanding of like, what was you know, how many people were there and how many people gave us their information and who completed a survey and who did all of these things and taking that data and having to then say, okay, like, what do we do in each of our our markets, you know, our geographies because we are a global team?

And from there, using it to say, okay. Like, we will do this and we won’t do this next year.

AI is also really great for those menial efficiencies.

I there’s so many Yeah. Like little things that we have to do when we’re creating slide decks. So we’re using we use Asado for project management and putting all those things together and saying it twelve times. And and I’m personally looking forward to just making us more efficient with AI. So it’s not taking out, like, the strategy part of our brain. It’s taking out the day to day part of our brain.

And so it’s using all that data, figuring out what you know, twenty twenty five was a lot of us trying new things. Legitimately, we talk about it and we were experimenting with the types of events we attend and the things that we’re doing. And we have said that to our leadership from the beginning of like, hey, we might not see results from some of these these events, but we have to try it once because then at least we know whether it worked or not, which is what we’ve always done an experiment or on the website too is like sometimes you just have to try it once. Someone comes up with this idea and you’re really skeptical of it, and you’re like, I don’t know if it’s going to work, and you do it anyway.

No. And so, you know, we’re doing that with events too. And being able to sit down and look at, you know, what worked and what didn’t work, what didn’t we get to do this year is all part of that planning process. And the last thing I would say is, like, personalization is really interesting when it ties from web to events in particular.

It’s not about custom content necessarily anymore. It’s more about, like, that relevant content, which is I think a place where people miss. They’d rather, like, say the person’s name and say exactly who they are instead of giving them something that makes them feel like it was designed for them.

So that’s really important to me is as we’re thinking about a website or an event in particular, like from the first touch point to the last that they feel the experience was entirely designed for them and not just like overly personal about them.

Yeah.

Well, what I appreciate is that we’re all in the same boat basically saying, don’t outsource your thinking to AI.

Use AI in order to scale, in order to take the strategy that you’ve built, and everything that you have thought about and have been creative about and now have this road map and this plan that’s rooted in customer strategy, use AI to execute on it, but not as a the one to build the strategy.

Don’t ignite your brain, I suppose. I think that’s take away.

Yeah.

Well, Sanjana and Erica, thank you so much for being here with me today. It was an absolute pleasure to hear all about experimentation and everything that you’re doing, and good luck.

Thank you everyone for attending and for watching us. And you can reach out to us, I’m guessing on LinkedIn, everyone. Is LinkedIn the good space? Yep. That’s where we’re at. So reach out to us on LinkedIn if you have any questions.

Until then, bye.