Note: This transcript was created using AI transcription and formatting tools. While we’ve reviewed it for accuracy, some errors may remain. If anything seems unclear, do refer to the episode.
Episode Trailer
Reuben John: If you were to look at where experimentation is headed, what are you most excited about?
Mahek Mahendra Shah: Users are now empowered to run complex tests easily. The entire experimentation industry — its trajectory has just started. People who have never planned CRO are moving into this, so how well we can help them build their plans fast within the first five days after they get onboarded — all those things will matter a bit more than what it was in the past.
I’m helping businesses optimize for their business processes. You work with all types of teams, whether it is marketing, sales, pre-sales, then whether it is engineering, doing podcasts like this. Experimentation is a tool to tell someone politely with data that whatever intuition was planned, it doesn’t look like it’s going to be successful — and this is the experiment, this is how it was run.
You want to introduce a feature which is a combination of engagement across Platform A plus B plus some activity or click, etc. Feature flags are the way to build custom segments and then release certain features to targeted audiences. VWO’s position, the kind of AI tools we have — whether it is in the creation flow, segments configuration, session recording, bulk analysis — a number of things in which AI is saving you a lot of time.
About VWO AB Tasty
Welcome to another episode of the VWO Podcast, recognized as a finalist in the CMA Awards 2025 for Best B2B Podcast. We bring you conversations with leading voices in CRO and experimentation to help you build better digital experiences and drive measurable growth.
This episode is brought to you by Wandz, our AI-powered optimization system that analyzes user behavior, identifies friction, creates ready-to-launch experiments, and surfaces actionable insights — so there’s less manual work for you and more strategic decisions for the business. Book a demo now to see Wandz in action. Now, back to the conversation.
Guest Introduction
Reuben: Hi everyone. My name is Reuben, Product Director at VWO, and welcome back to yet another episode of the VWO Podcast. I’m really excited today for this conversation because our guest is very well known to us. He’s Mahek Mahendra Shah, Director of Product Management at VWO — a colleague and teammate. Many of us at VWO know him for the role he has played in evolving key parts of the product and the product experience over the many years here, especially around VWO testing and onboarding for new users. Over the years he has worked across different areas of product and digital experience, giving him a practical understanding of how experimentation programs evolve, how product teams make decisions, and how customer experience influences product thinking.
In this episode, we’ll talk about the learnings from that journey, what he’s currently focused on at VWO, and where he sees experimentation and product innovation headed next. Let’s get started. Mahek, thanks for joining us. Excited to have you here and finally having this conversation together.
Mahek Mahendra Shah: Sure. Thanks Reuben, thanks Ashley for organizing this, and I hope I do justice to your questions.
Reuben: For sure. We’re looking forward to this. Plus I get a chance to put you on the spot — doesn’t happen often.
Mahek Mahendra Shah: Yeah, fair enough.
Icebreaker
Reuben: Okay, the first section we have is an icebreaker round. Light, not necessarily related but somewhat related to the topic. The question is: what do you love most about working in the CRO industry?
Mahek Mahendra Shah: I think it’s the same with my previous experience and this experience — the variety of businesses and their organizational processes. It’s just exciting to see how different industries operate differently: different business margins, different psychology, different ways of execution. It’s always fun to be in an industry where you have all types of clients.
And CRO is optimization, right? Optimization is basically natural for human beings — we optimize every day. To go from place A to place B, I can take the metro on a traffic day, I can take a bus on another day, I can take a bike because I know the car lanes will be slow. So we optimize every day. The CRO industry is just natural for us.
Reuben: Interesting perspective, Mahek — starting off with a very philosophical note. Optimizing versus, I think the counter to that is satisficing. Not all humans are optimizers, but you’ve spoken like a true product manager — aspirational for everyone. Thanks for sharing that. Let’s get started with the more interesting questions then.
Conversation
What Keeps Experimentation Genuinely Interesting
Reuben: So Mahek, you’ve helped evolve products at VWO — especially around testing, insights, and feature experimentation, which is quite a range. Before we get into all of that, what is it in this space that keeps you genuinely interested?
Mahek Mahendra Shah: I don’t see experimentation as a different industry. To me, it is just part of product management. And essentially, product management is ownership of products, which means you’re running a business. So it’s a natural part of the business you conduct on a daily basis.
What is interesting about it is the scope. You are looking at all angles of a business and all stakeholders. You work with all types of teams — whether it’s marketing, sales, pre-sales, engineering, post-sales, onboarding, retention, PR teams, even doing podcasts like this. There is not a single department that isn’t part of product building. And in every case you are working to optimize your products and maximize business value for your customers.
It’s just product management to me. And in this case, I am helping businesses optimize their business processes.
Reuben: Yes — optimization within optimization. You’re building a product that helps other people optimize, so it’s like product management inception for you right there.
Mahek Mahendra Shah: Probably. And there are some other points too. As a product manager, you typically have some ideas, your management has some ideas, your team members have some ideas. What happens when your organization grows big — everyone has ideas. Which ones to take? That becomes a big challenge. How do you break those human barriers? You need a data-backed process. And if you look at it, humans can make errors. Data is something humans interpret. So by running this CRO process, you have raw data available, and the decisions you take, people are more likely to trust them together rather than just someone saying, “Hey, do this, this is better.”
That need for building trust is easier when you run CRO programs that have very good data backing and are properly configured.
And there’s a famous quote I generally put in my presentations. It’s from Professor Richard Feynman: “It doesn’t matter how beautiful your theory is — if it doesn’t agree with the experiment, it’s wrong.”
Reuben: Absolutely. Numbers don’t lie at the end of the day — that becomes the bottom line.
Patterns in Onboarding and What They Signal
Reuben: You spoke about enabling different businesses, and of course businesses exist at different stages — both as organizations and in terms of experimentation maturity. Given that you’ve helped onboard many customers and businesses onto our platform, what are some of the interesting patterns that repeat, and what do those patterns tell you about where that business will end up?
Mahek Mahendra Shah: So in this also, there are some things you can say. A one-person company and a company with a thousand people will both run CRO programs in very similar ways.
But we’re in an era where traffic is becoming scarce. AI is delivering answers without landing you on ten different websites, like it used to. Every single visitor interaction counts more than ever. Everyone is looking to optimize. It’s no longer a luxury — it’s an absolute requirement for survival. And it’s not just for small or big businesses. It’s for everyone.
Even if you look at businesses, both small and large, when it comes to their CRO strategy, they will put similar types of goals in their objectives: “I want to increase my conversion percentage by 5% or 10%.” For a company with $5 million, 5% is different. For a company with $500 million, 5% is different. But they’re all looking at improving some key metrics at a certain percentage.
Every business is different — even businesses in the same industry, even in the same lane, will operate differently. But there will be some common themes: people should get to the information they’re looking for as early as possible; people should be able to check out easily; people should be able to get help whenever they’re stuck. Some themes are very common across every type of e-commerce store or website.
We are fighting for visitor traffic much harder than ever right now. And CRO is very natural in almost all big organizations these days.
Reuben: And I think a lot of our listeners will resonate with the fact that top-of-funnel is generally drying up, while traditional channels are getting more expensive. So the folks who do come in drop deeper into the funnel — you can’t avoid optimizing for that. But just to follow up — are there any cues that businesses display during onboarding that tell you this is going to be successful, or perhaps not?
Mahek Mahendra Shah: It depends on the persona of the users who are onboarding. When different personas sign up, their objectives are very different.
A developer who’s been asked to check out the product — his objective is: will I be able to integrate the SDK, how will I integrate the smart code? If it’s the marketing manager, he wants to preview how conversion percentages come through and whether he can make a report out of it. If it’s the C-suite level, they’ll look at the overall program: how do I track health? How do I show ROI at the end of six months? If I’m spending $4,000 per month on this product, can I show a 2% increase in my overall revenue metric, which comes out to $90,000 or $150,000 a year?
Every persona comes to the same product with a different objective. That’s where some things can be generalized — everyone has to reach a small aha moment. But within the first week, we get cues of what kind of person is checking out the product, what speed they’re using it at, and so on. Based on that, we typically have different persona-based journeys that get triggered. Some are completely product-led journeys, and some could involve inserting a sales or pre-sales representative early — because C-suite level people may not be able to spend enough time on a product to check it out fully themselves.
When Customers Use Features in Unexpected Ways
Reuben: Thanks for sharing that. And to take this forward — when we build a product for such a wide range of customers and personas, surprises are inevitable. Is there a feature that you built to solve one problem and customers ended up using it for something you never intended? And what did that teach you?
Mahek Mahendra Shah: This has happened many times. You build something to provide an audit trail, and they use it for commenting and chat between teammates. You build something for sharing with a team member, and they directly use it for company presentations. It’s just, again, which persona is using which feature at which point in time.
Almost all big features have multiple usage perspectives. We do user interviews where we shadow clients, and we have session recordings where we check how people use the product — specifically the onboarding pages in the first 30 days. We are surprised every fifth day with a new pattern. Customer to customer, their context, their moment, their time, their usage — all of it combined gives us a different perspective on the same feature.
One more thing that adds to this is the maturity level in terms of experimentation — of the users and the organization. You and I, having heard about CRO for years, are more mature when we use the product. We’re not discovering things; we’re looking at implementation and how it’ll benefit us. But there are a lot of marketing professionals and fresh developers who will check out the product from a beginner or aspiring perspective, looking to enter the optimization industry. That maturity level also decides how you’re going to use the product and how fast you’re going to reach that aha stage.
You can make general reports, but at the end of the day, you need to personalize per person. You need both — a generalized overview to present to management, and taking care of individuals and their journeys, handholding them properly.
PLG in Action: A Real-World VWO Example
Reuben: Got it. You touched upon personalization there. Moving on — this is something you’ve driven here at VWO. Can you talk about a PLG initiative that you’re proud of, what you did, and what made it work?
Mahek Mahendra Shah: PLG initiatives are something I’ve taken up in almost every organization I’ve worked in, including my startups. It’s the most impactful thing you can do when you know you can guide a user all the way to a certain point until they really need your help.
PLG motion for B2B and B2C are typically different. Let me give you a B2B example from what we did here at VWO. This was one of the first projects I took up when I joined.
We have users signing up — we didn’t know what they did by the end of their free trial period, whether 30 days or 14 days. What we did was break down the ideal journey for a user: how, for a given persona, they should be doing a set of activities to reach the aha moment if they’ve come to an experimentation product.
So they sign up. First thing: tell us what website they’re planning to optimize — they add the website. Next: they create a campaign and try to make some changes. Do they understand that for experimentation, they need a control and a variation? Are they writing a hypothesis for why they want to make this change? And then more advanced configuration — whether the test should run for all traffic or a subset.
To start a campaign, there’s one key hurdle — they have to add our smart code to their website. If someone takes the effort of adding our code to their website, it’s a very strong signal that this person is interested in experimentation. They’re not just here to check things out. They’ve put the code on one of their websites, which means experimentation is something they’re looking to implement. So for us, the minimum strong signal for someone to become a highly qualified lead is that they add our smart code and verify it’s working. The same applies to SDK integration for feature experimentation — we have similar signals there.
Once they’ve put our code in, run an experiment, and started getting traffic — if there are serious people looking to get optimization results within the free trial, you would see a campaign declare a winner or a loser within that period. Now that’s a super strong signal.
So say we have 1,000 users who signed up in a time period — imaginary numbers, but say 50 only added a website, 150 created a campaign, 150 started the campaign, 120 verified the smart code, 160 got their campaign to at least 1,000 visitors, and 80 of them got a winner or loser declared. At the end of 30 days, where are all these people? What bucket are they in? This is what we defined as their funnel.
Now, anyone can get stuck at any one of those funnel steps. What we did was: when someone was stuck on the same funnel step for more than a day or two, we sent them an email saying, “Hey, you’re at this step right now. Typically, what other people do when they’re stuck at this step is these set of points.” We used our session recordings and user interviews extensively to come up with a standard set of four or five recommendations for each funnel step — to give them the nudge to move to the next one. And at any point where we observed they needed help from one of our sales or pre-sales reps, that was also gently introduced in the email itself.
When we started, we had a generic welcome email that just said, “Install our smart code. Best of luck.” No other emails. When we implemented this, we had close to 18 different emails across different parts of the funnel. Currently that number is around 38 because we’ve added more products as well. Each email has been relevant. We’ve seen improvement in people moving from the low-intent funnel to the high-intent funnel, which is a big number — specifically because organic traffic is a challenge, but we’ve been able to improve our conversion rates at a good enough pace to still maintain good organic growth.
That’s one of the PLG-led initiatives we took up. It involved a lot of our own tools — user behavior insights, session recordings, user interviews, conversations with our pre-sales consultants, and looking back at customers who became paid and asking them what we could have done better. All of that combined, together with nicely designed clear emails with next action points — it worked well. One of the success stories is also on our website if someone wants to read it.
Reuben: Perfect. VWO using VWO is a very interesting case, and that’s what Mahek has done there to great success. Thanks for sharing that.
What Organizations Miss When Testing and Insights Don’t Connect
Reuben: Moving on. A lot of teams run behavioral insights and experimentation as two separate streams, consumed in different ways. In your experience, what does a business really miss when testing and insights don’t feed into each other?
Mahek Mahendra Shah: This is a fundamental challenge in many organizations — whether the person or team responsible for CRO is capable of using not only insights but also testing alongside them.
Let me give a real-world analogy. Let’s talk about three sets of people: general public, activists or NGOs, and politicians. The general public will tell you: “There is a pothole here.” “There is an open drain here.” “There is no cleanliness here.” That’s the majority of people in any organization too. Then there are activists who will tell you: “This is a pothole, but it needs to be fixed with concrete.” “This area needs to be cleaned daily.” That’s the second set — people who identify the problem more specifically. But then there’s a third set — the politicians. They know the problem, they know the activists, and they’re supposed to execute and solve that problem.
If your organization is running only insights, you are basically just discussing. You’re not taking action. It’s just information you can get, document, and present to higher management — and there’s a chance it’ll become action eventually.
But if you can identify the problem, document it, document proposed solutions, set up a strategy to measure it, and roll it out successfully — that’s the entire CRO cycle. If an organization is doing only one part in silos, it’s not a great thing. Unless your organization is very small and one person can do it all, you need a good set of team members who look at this end to end — all the way from analyzing and capturing problem points, to capturing solution variations to improve success metrics, to eventually rolling it out such that you can see a revenue impact number.
When Data Becomes a Flood: The Tipping Point
Reuben: Very insightful. And on the topic of data — numbers don’t lie, but there is a flip side. There are instances where teams are sitting on behavioral data, analytics, clickstream, everything — they know what the user is doing, where, and how — but they just stop converting insights into action and start drowning in observation. Where does that tipping point sit, and how do you pull a team back from it?
Mahek Mahendra Shah: For this, I go back to first principles. Whenever you’re doing data analysis, first divide it into two simple things: raw data and derived data. Raw data is the number of people in your organization. Derived data is: five of them are managers, 16 are engineers. Any report made on raw data will be very trustworthy. Any report made on derived data will need further scrutiny.
Second, there’s the complexity of interpretation. Present the same data to two people, and they’ll talk about it differently. Let me give you a recent example. We had state elections in one of the states in India recently. The opposition had an 8% vote share swing in their favor. The ruling party had a 6% swing away from them. So there’s a 14% total swing against the government. Now, the opposition will say: “We’re just 4% away from overtaking the ruling party, and we had a 14% swing in our favor.” The ruling party will say: “We’re just 4% behind.” Same data, two completely different stories. Data will not speak for itself. There is always a storyteller who will frame the data the way they want it heard.
So you have to be very careful when presenting optimization results. We have statistically significant results, and we have results that show positive signals but without statistical confirmation. At the end of the day, it’s always the end user who decides how the data story gets told.
Volume of data is interesting too. Recently we had a prospect — a very big US organization operating for more than 100 years. In the first call they said: “We have 100 years of data but we don’t know what to do with it, and we want to start running CRO programs.” They were a company where business was overflowing, and they had no time to sit and analyze what could be optimized. Now that the competitive landscape has changed in the last five years, they’re looking to optimize.
On the other side, you have organizations that were started very recently, but the founders come from a very strong CRO background and are moving into this industry with CRO as one of the core levers for growth from day one. So whether you’re sitting on mountains of data or have none at all, it’s all relative. High data or low data doesn’t define where you start. What matters is the trigger and the intent to act.
Reuben: Just to summarize — there’s an aspect of trust in the data and how we interpret it. And then it really doesn’t matter on the volume of data, but rather the trigger. One of those triggers could be competition, tighter margins. And the thing I’m also taking away is: you’d rather start on that journey earlier than later, because one way or another, that maturity is going to hit you.
Mahek Mahendra Shah: One more point I can add on results interpretation: you typically won’t wait for perfect information or a perfect result to take a decision. Many times you run experiments to validate a hypothesis where you have a good intuition. And even if you see signals that the intuition is being backed up sufficiently by data, you may not need to wait four more weeks for statistical significance before moving ahead. Sometimes people do take those decisions early to back up intuition — especially when their intuitions are generally good and have been validated by data in the past.
The reason sometimes people take these early decisions is that time is your most expensive currency. You can’t run experiments for months. Sometimes you have to take the learning from a two or four-week experiment and take an early decision.
Reuben: Something I’ve heard from some of our customers as well. It’s a double-edged sword — there’s the risk of peaking and the statistical significance you lose. But at the same time, tools like Smart Stats that VWO has, which is quite exclusive to our platform, help simplify this and make it more democratic.
Feature Flags and the Growth Angle
Reuben: Moving on — and this is personally my favorite topic because it’s part of feature experimentation, which is something I lead. Feature flags have become one of the most powerful tools in a product team’s hands. Traditionally it’s been a way for engineers to safely deploy code, but now it’s grown into something that product managers, data science folks, product leaders, business leaders, and marketers all want to control — because it becomes the control panel for your entire product. Bringing in the growth angle: how do you drive growth when you’re targeting these very diverse personas, each with their own objectives and intentions?
Mahek Mahendra Shah: Feature experimentation — with the pace at which technology implementation is becoming easier, experimentation will be a necessary add-on in the near future.
From an engineer’s perspective, you want system stability. Feature flags are the right way forward in that direction. You want to roll something out gradually, or do a 100% rollout, or roll back in one go — you do that with feature flags very easily. And I don’t need to give you examples for it. Literally, every three to four weeks, we see a very big organization screwing up a rollout and impacting billions of dollars in business.
But you look at product managers and marketers — they’re looking at feature flags from a targeting perspective, segmentation perspective, quick rollout perspective, and experimentation perspective. Their objectives are a little different from engineers.
One of the most popular examples for feature experimentation that I keep giving in my presentations: any product offered on desktop, mobile, and multiple platforms — TVs, smart TVs — feature flags are essentially fundamental. If I’ve browsed something on mobile and I open desktop, it should reflect in exactly the same way. Hey, you just watched “Durandhar” and paused halfway through on your phone — when you go home and open the TV, it shows “last watched” and picks up exactly where you left off. Your login is saved across devices in one place. And if you want to introduce a feature that’s a combination of engagement across platform A plus platform B plus some click activity, feature flags are the way to create custom segments and release features to targeted audiences.
That’s something marketers and product managers are doing extensively. And feeding feature flags into your MCP models or AI models to generate a set of experiments is something that is the future. System stability remains the biggest use case — no doubt. But personalization is the second biggest use case across the industry for feature experimentation.
Reuben: Absolutely. The point that stood out to me is: it is inevitable. Cross-product platform experiences are everywhere. The way things are going with AI, AI-generated features, AI code — there’s a speed aspect, and there’s an aspect of non-determinism. You need a way to control that risk. Inevitably everyone’s going to want to be a part of this.
Mahek Mahendra Shah: And to add to that: deployment of a feature is an engineering decision. Releasing the feature is a business decision. Rolling back the feature is a business decision. Feature flags give you the simplicity to do all of this and time it exactly how you want it — that’s the essential business value of feature experimentation.
Reuben: Absolutely. Decoupling release management from deployment is so key today.
Experimentation Culture: How Many Companies Are Actually Living It?
Reuben: Coming back to the main section. We keep hearing about experimentation culture in the market. OpenAI acquired Statsig — people are talking about experimentation being core infrastructure. In your honest view, how many companies are actually living that? And what’s still standing in the way for most of them?
Mahek Mahendra Shah: I don’t think the industry has adopted it in the way we would have wanted — though of course that’s from our point of view. Ten to twelve years back, maybe 5% of organizations were doing experimentation. Today, 80 to 90% of companies know very well what experimentation is. 40 to 50% of big organizations have some plans or mention it in their annual reports.
For smaller organizations, growth is their biggest hurdle, and experimentation is a sidekick. “I need to sell 10,000 more units. I need 1 million more traffic to my website, for which I need to spend in ads.” But as soon as you grow to a certain revenue level — maybe $5 million or $1 million annually — improvement in conversion percentage starts mattering much, much more. Any small percentage increment is big revenue. That’s where big organizations have experimentation as an item within their annual strategy.
Whether they’re implementing it wholeheartedly is another matter. Organizations where management has more technocrats, where technology has been adopted fast, those tend to embrace it more. And it’s not about age — we’ve seen organizations led by 75-year-olds that adopt technology very fast simply because the management is pro-technology, pro-conversion numbers, and business-minded. Someone brought up in the technology era may find it more natural, but people on both sides can get there.
And for bigger organizations, the fear of failure is less. They’ve grown on the back of thousands of failures. Experimentation is just one more way they can keep testing while showcasing winning rates. Sometimes in these organizations, experimentation is also a tool to politely tell someone with data that the direction they suggested may not work — including telling a CEO. You can’t go to a CEO and say no directly. But you can run an experiment and say, “Here’s what the data shows.”
So bigger organizations can use experimentation programs to validate before rollout across different geographies, do gradual rollouts, control releases. The culture is dependent a lot on the people managing and running operations and how they take it forward.
We ran an experimentation maturity survey recently of organizations using VWO. We’ve seen a drastic change in the numbers in the last one and a half years. Organizations that were at a beginner level — 80 to 90% have moved to a more progressive experimentation approach within the company because they’ve seen results. As soon as you see the numbers, everyone jumps on the wagon. A 2% to 5% conversion rate improvement for a $100 million company is millions of dollars — by spending just $5,000 per month, you’re getting additional millions. That math speaks for itself.
Reuben: Absolutely. And I think the point around using experimentation to challenge the CEO with data is a very important one. In a way, experimentation culture correlates to the organization’s culture and vice versa. More experimentation may mean a flatter, more open organization. Less HiPPO culture — Highest Paid Person in the Room culture. And companies that use experimentation may actually end up changing the culture of the company itself. That’s an interesting loop.
AI and Experimentation: Speed, Governance, and the Human Element
Reuben: The next question involves AI — you can’t have a podcast in 2026 without it. AI is now making experimentation much easier, compressing the cycle from setup to results. Accessibility is improving, speed is increasing. But enterprises are now telling us this is creating new problems — governance, and the human element. Teams are not able to keep up. What do you think about this trade-off? And how does VWO bridge that gap today?
Mahek Mahendra Shah: AI is the Pandora’s box that has opened recently. It’s a step in the evolution of technology. The kind of opportunities it has created for businesses is mind-blowing and amazing almost every day.
From an experimentation perspective, the fundamental change AI has brought is that time to experiment has dropped drastically. Something that would have taken you seven to fifteen days to figure out — you can now do in a day or a few hours. What to experiment, how to experiment — AI is helping increase speed. Personalization — AI is helping enormously there too.
The simple fact that you can experiment faster also aligns with the “fail fast” mindset. If you want to try something, you can fail fast. Generate enough traffic, run the experiment, fail fast — you don’t need to wait a long time. Earlier, if you needed four people to plan an experiment, with AI, two people can plan, run, and analyze an experiment. So in terms of cost savings, from an executive level position, yes — AI is adding to the ROI of tools like ours as well.
Now, what is the output generated by an experimentation program? I think that remains the same. You ran an experiment and got a result in the past. You run an experiment and get a result in the future. What has changed is the time to run it, to configure it, to present the findings. But at the end of the day, whether you want to implement the result is still a business decision. AI is not going to be accountable if something goes wrong. The people running the CRO program will be accountable for whatever the output is. AI can assist in configuring and running, but the decision on whether to proceed still largely remains with the CRO program managers.
And from VWO’s position, the kind of AI tools we have at every step of the flow — whether in the creation flow, configuring segments, session recording bulk analysis, heatmap analysis — AI is saving enormous amounts of time. Things that would have taken hours or days can now be done in minutes or less than an hour. Time savings is the biggest offering coming into this industry from AI. Decisions at the end of the day — the CRO program manager has to own the outcome of the experiment.
Reuben: Just to add to that point — from my experience knowing the platform, we do add that governance layer which puts guardrails and process-adherent guidelines in place so we don’t exceed limits. Security is something deeply embedded in our systems. Combining everything in one place, with all your AI enablers on the platform plus that governance layer to keep you on track — that’s powerful. It’s the answer to the problem, which is: things are getting really quick, we need to fail fast, and we need to react.
Mahek Mahendra Shah: In organizations, audit is a key aspect. Using a tool like VWO, almost all key actions are audited and available for scrutiny later on. Security is something we have continuous certifications around. We do annual reviews, audit reviews are done every six months. We have certifications from Europe, from the US, all North American certifications required for Fortune 500 companies. And our turnaround time in case of any audit-related matters is very good from our perspective.
Reuben: Correct. And that feeds into the whole build-versus-buy piece as well. I’ve been on the other side where we tried to build it out internally. When it comes to these nuances — the engineering effort, the maintenance overhead — it becomes a much more complex piece. And then you have a platform that’s already solved for that.
Mahek Mahendra Shah: The build-versus-buy decision is very much a management-level decision. Do you want to build something that isn’t your USP, that isn’t something you’re going to sell — when there is something available in the market that is readily usable and validated by thousands of paying customers? That’s a decision management needs to take. If you have spare resources and want to build internally, that’s up to you. If you want to use something that’s available and certified — great.
Reuben: Especially tying it back to the AI piece — velocity of releasing features and products has become very high. There’s a notion in the market that SaaS is dead, but I strongly feel otherwise. It’s hard to build this out and maintain it going forward. The value-to-effort ratio is skewed towards buy for most cases. And AI just doubles down on that, given the speed and governance aspects we just spoke about.
Where Experimentation Is Headed: Excitement and Concerns
Reuben: We’re nearing the close of our main section. Looking at where experimentation is headed — both the tools and the mindset — what are you most excited about? And at the same time, what concerns you the most?
Mahek Mahendra Shah: The most exciting part is that users are now empowered to run complex tests. In the past, our customer success managers were discussing simpler tests with customers. Now they’re going to be discussing much more complex tests because it’s possible to run them faster, and setting them up is easier too.
What am I worried about? Honestly, not much right now. When you give someone an opportunity to test, they will test — that’s natural human tendency. More organizations are going to test for a good amount of time before things settle down. We’ve just started. The entire experimentation industry — its trajectory has just begun.
But we do have to manage a lot of expectations. People who have never planned CRO are now moving into this. So from a product-led growth perspective, how well we can help them build their plans fast — whether it’s a Gantt chart within the first five days after they get onboarded — all of those things will matter a bit more than they did in the past.
Reuben: I think that’s a good problem to have. We’re having more folks who will use this, but guiding them the right way — which I’m sure the AI initiatives your team is building will help with. And on the excitement side — complex experiments. That’s where the frontier is.
Mahek Mahendra Shah: At the end of the day, I’m an engineer. And anytime complexities come and variables have to be analyzed, that excites me. That’s why it’s exciting for me.
Reuben: Absolutely. Just recently a customer reached out saying they’re going to use Claude Code to build variants and do in-house tooling to support testing — creating N number of variations using their coding agent. And the way we pushed back on that was: you still need to build the infrastructure to support these variations, to run the experiment, to track the results, to ensure statistical significance, to ensure you’re using the right methods. Is that something you truly want to own, even if one part of it is simplified with AI? The platform itself becomes the experiment infrastructure. That’s a great point.
Break
Reuben: Okay, before we go into the last segment — what’s your favorite travel destination and why?
Mahek Mahendra Shah: Favorite travel destination. I think for me, mountains or the desert — both. The mountains because there are so many challenges. The desert because there is nothing and you have to survive. They are extremes in their own way.
My native is in the desert, and every time I go to the mountains, I’m just surprised at how totally different every single mountain range and every single trek is. The Himalayas are my favorite destination. Anything related to the Himalayas has defined our culture for a really long time. My mother’s village is on the banks of the old Sarasvati River, which originates in the Himalayas. It’s a thousand kilometers away, but we are directly connected to those mountain ranges. So yeah, those.
Reuben: Very cool. Thanks for sharing that, Mahek.
Rapid Fire Round
Reuben: Okay, last section — rapid fire questions. Short but spicy answers. Let’s go.
Reuben: One CRO metric that you wish people would stop obsessing over.
Mahek Mahendra Shah: Click on Add to Cart.
Reuben: Interesting. One thing your non-industry friends still don’t understand about your job.
Mahek Mahendra Shah: Experimentation. Many, many of them don’t understand it.
Reuben: Fair. One thing that AI will completely take over in the next two years — please don’t say our jobs.
Mahek Mahendra Shah: Children in the age group of seven to ten will stop doing group study, because AI is their new buddy.
Reuben: Interesting. If not CRO, what other profession would you have chosen?
Mahek Mahendra Shah: Hardware manufacturing. Always exciting. And remember, hardware manufacturing is also undergoing a transition because of AI. We’ll be able to manufacture cool things much faster.
Reuben: Absolutely — the next frontier of innovation for sure. If you were starting a career in CRO today, what is the one thing you would do differently?
Mahek Mahendra Shah: Looking at the outcome of a CRO dashboard early in your journey helps you figure out where you’ll go in this space. So looking at sample dashboard outcomes and reading success stories in the beginning really helps you.
Reuben: Got it. Last question — any final thoughts or messages you’d like to share with our audience before we conclude?
Mahek Mahendra Shah: Yeah, I think — have a good day.
Closing
Reuben: Okay, that’s short and sweet. With that we’ll wrap up our episode today. Mahek, thank you so much for joining us and for sharing your experiences, perspectives, anecdotes, and travel stories. I’m sure our listeners took away a lot from this conversation — not only from a standpoint of experimentation, but I’m sure our product managers and CRO specialists got a few golden points they can take home today.
To everyone who tuned into this episode of the VWO Podcast — if you enjoyed the conversation, make sure you subscribe so you never miss an upcoming episode. We’ll be back soon with more conversations featuring leaders across experimentation, product, growth, and digital experience. See you in the next episode.
Mahek Mahendra Shah: Right.
