Embedding Experimentation In Company Culture

0:06

Bhavya: Hello, everybody! Namaste! Welcome to ConvEx. This is VWO’s very own in-house online Summit, where we are talking about all things experimentation. Of course, I don’t need to introduce VWO. It’s the world’s leading consumer insights discovery and experimentation platform. Companies are using us to figure out what are their conversion bottlenecks and removing them through forward-looking A/B tests done through our platform. Just give it a spin and tell us how you feel about it. I’m really excited to have onboard Paras. He’s the founder of VWO. My name is Bhavya. I lead account-based marketing here.

0:47

This chat is a very freewheeling conversation between me and Paras about what makes a conversion-centric organization, a conversion-first organization, and an experimentation-driven organization. Paras, through all his years of experience, is going to talk about how to distill an experimentation-first culture. It’s a topic that is dear to both of us, and I believe you will have a lot of enriching insights once you are out of this stock. There is this LinkedIn group that we have in which all the subscribers are hanging out. They are networking. The link is right here on your screens. Just go and check out for yourselves – what is the chatter up to?

1:27

Paras! Please take it away. 

Paras: Hi Everyone! So today, I’ll talk about embedding experimentation in the company culture. The reason I picked this topic was that throughout my experience as founder of VWO, over the last ten years, I’ve seen many companies rightly embrace A/B Testing in terms of helping them increase the conversion rate.

1:52

And, after all, that is why most companies start with A/B Testing. They want their signup rates, conversion rates, and any of the business metrics to go up. But, I’ve come to understand by learning from many of our clients that there is sort of a larger value that can be derived through A/B testing, which is transforming the entire company culture into becoming more experimentation-focused, so that it moves away from just being about optimizing marketing metrics to everyday conversations of people and team members talking about A/B Testing various different things. So, let’s start by understanding why am I well placed to talk about this. Bhavya briefly mentioned about VWO. But, my personal involvement with VWO obviously has been from day 1. I have seen more than 1 million campaigns run on VWO with thousands and thousands of clients across so many industries, and even from a business site’s perspective, I have personally interacted with a one-person company all the way to Fortune 500 companies. I have had the privilege to see experiments being run in a variety of different contexts, industries, and business sizes. And, in this presentation, that is what I am going to share with you, right from the very basics for someone who’s just starting out with A/B Testing to people who have really mastered the art of experimentation. What does that journey look like and how in your organization tomorrow you can go back and talk to your peers, colleagues, and even your boss and really discuss – how do you move from where you are today to where you want to be in terms of building an experimentation culture.

3:50

And Bhavya, as I said, feel free to ask any clarifying questions you have on behalf of other viewers of this presentation.

4:01

First, let’s talk about why to focus on culture at all. Culture is one such word, which has been used in a variety of contexts in many cases, even misused as well. I mean, before just ‘why focus on culture’, let me at least define culture. So, culture is not really having to say fancy posters in office or going to maybe some strategic off-sites and so on. Culture, to me, is what decisions are being taken by team members and people in the company without the oversight of their superiors or their bosses. Culture frankly in one line is simply, “The decisions that are being made by a company in their everyday operations.” By experimentation cultures, by just the extension of that, it is really about when people are talking in an office. They are trying to make decisions.

5:01

Better than making decisions in an arbitrary way, through a very opinion-driven way or whether they are really embracing the spirit of A/B Testing experimentation in their everyday decisions.

5:13

So, these decisions can go right from very small things such as, “What should we have for lunch,” to the really big decision strategic decisions such as, “Should you open an international office or not.” And, embedding an experimentation culture in all those decisions just means that the way conversion rates improve through online A/B testing various different business metrics will also start getting improved by focusing on that culture. Experimentation culture really means you have to invest. Let’s get back a little bit. 

Imagine, you are very enthusiastic about A/B testing. You purchase a very expensive subscription of a product or a service or maybe a contact agency to help you with conversion optimization, and initially in all the enthusiasm you make a lot of time and monetary investment. I have seen with a lot of clients that just that is not enough. A tool can tell you that A is better than B or B is better than A. Similarly, an agency can tell you that. But if internally, the stakeholders in your company are not prepared to accept that they could be wrong then no amount of investment is going to help you reap the benefits of experimentation because people will reject. If their mindset is not prepared from an experimentation point of view, they will reject the results that are not aligned with what they have been believing earlier. I have seen this multiple times. The highest-paid person’s opinion also prevails after an experiment is done because they haven’t fully bought into the culture of experimentation. And, as I said without this culture shift in your organization after initial enthusiasm, things will just go back to open enrollment changes, where people will just do whatever they want to even after they have data or even after they have A/B test results. So, that’s why building a culture by laying the right groundwork in your team is very important.

7:30

It takes time and effort, but without it, only initial enthusiasm will not cut. It will be back to square one.

7:40

Just to give you a flavor of what the epitome of this experimentation culture looks like. There are some companies that have absolutely nailed it. This is the level I’m talking about. So for example at Amazon, any developer can launch an A/B test anywhere on the stone without permission. Just think about it for a minute and contrast it with how things might be happening in your organization. Does your organization have a separate A/B testing team? Maybe you don’t even have a separate testing team, and the marketing team itself does the A/B test. But, can someone, say in the engineering team or the sales team or even in the HR team, can they suggest any A/B test or can a developer when they have a good idea. Do they have to go to the product team to get their feedback and then really start developing that feature?

8:35

So, there is so much complexity usually organization even an idea across that the best of the experimentation cultures like Amazon, they allow any developer to just launch an A/B test because they understand that an opinion is just an opinion and real data can only be actually tested. Similarly, at Booking.com, anyone in the organization, including the CEO and including someone in the administration department, who may have nothing to do with the testing pipeline. They still have the ability to feed their testing ideas into the testing pipeline. How amazing is that? This really shows how right from the top the entire organization understands that these great experimentation ideas can come from everywhere. On Facebook, thousands of experiments are running every day.

9:30

So, the version you see, Facebook is very likely different from the version I see. This means that across the entire Facebook at any moment of time, and there are so many tests running that it has to be decentralized. There won’t be anyone who is keeping track of thousands of experimentations. To come to this stage, Facebook must have had invested so much in the underlying infrastructure and also the mindset. And, the last example and perhaps this is the most popular one, the team at Google is so obsessive over A/B testing that at one point of time they tested 11,000 shades of blue to see which shade of blue gives them the maximum click-through rates in organic search results. Now, that’s the level that they are willing to go to in order to really drive your testing culture in their organization.

10:24

Bhavya: So, my first question comes here. Clearly, what you have mentioned on this slide is aspirational for a lot of companies. They have aspired to reach here. They’ve read all these stories about the names that you have written booking Amazon. I personally wrote a blog about how booking things about experimentation, and I have had a lot of subsequent conversations with the customers with a lot of our customers and agencies around how to amplify that culture. They believe that it’s not that the merits of experiments are not dawned on to them there. They know very clearly about the impact that experimentation has on overall metrics, not just acquisition but the entire AAARR framework. The revelation here is that it’s more of a change management problem. All right, it’s not a problem of understanding how important it is. It’s about how to bring everybody up to speed with what the new culture is all about and how to get everybody tuned in and zoned into the benefits of having sustained experimentation with high velocity. 

How do you think companies should do that? And, what are some successful examples that you have seen of our customers may be, for example, that have deployed this very very well in their organization. 

Paras: Sure. So, great question Bhavya and with these Nobles what I didn’t mean to convey was that say next month next quarter or even next year, an organization can really claim to be someone like Booking.com or Google or Facebook. At least when it comes to experimentation culture. I think where these organizations are today, is a result of years of continuous investment into that mindset.

12:16

So, it will be important to understand that what we’re looking at is “The Cutting Edge” of how what experimentation cultures look like. To get to this stage would require a significant amount of time and effort. Rather than just fishing for this and the entire presentation, my rest of the presentation, is frankly, about that. It’s about identifying where your current experimentation efforts stand and then really mapping out some sort of a road map to get to the level where these cutting-edge companies are. So, it cannot be done in one day, one quarter, or one year. If the importance is understood by customers, then a very specific roadmap has to be planned so that one after other different roadblocks are removed and the culture moves towards such cutting-edge cultures. To get to that the whole presentation is about how do you map that roadmap.

13:17

So that, in fact, the next slide is about the roadmap. In my experience, I have seen the customers that I have interacted with, I mean, obviously, it is a continuum boarding the interest of clarity. I could deliver or categorize these into four stages of experimentation culture – Stage 1, Stage 2, Stage 3, and stage 4. 

Stage 1 is really about companies with no experience or history of testing at all. These are the companies who heard about testing experimentation somewhere, and they’re very enthusiastic about trying to do testing for the first time. 

Stage 2 is about once you have purchased a subscription of a product like VWO or when with a free product or you’re just entering into doing your first initial A/B test. Usually, I see that people do like simple low hanging fruits and they occasionally do A/B test, because usually the discovery of the importance of A/B Testing is done by a department like a marketing department. In addition to their day-to-day responsibilities, they start a be testing some of the things in addition to whatever they’re doing.

After stage 2, in Stage 3, companies realize that even though occasionally A/B testing is better than no testing at all. They cannot rely on someone thinking let’s do a test today or someone coming across an improvement opportunity in a very random fashion.

14:57

So, Stage 3 is really about converting that occasionally A/B testing and doing more investments into people and processes so that conversion optimization runs as a program in the company, which means that just like there are targets in sales, there are recruitment plans, there are financial plans, and so on. Conversion optimization itself becomes plan in a different way for next quarter or next six months to map out. What are the tests that you’re going to do? What resources are required for it and how frequently you are able to run that. Then, once you do that, the state 4 is the stage that we were talking about in the previous slide where companies like Booking.com, Google, and Amazon come into play. In addition to being process-driven, this really becomes continuous experimentation across all business functions. So, it is really “The Cutting Edge,” where anyone can run any tests, and that is the culture that is evolved from Stage 1, 2, and 3. This is not just limited to a website or even marketing. This seeps into many other business functions.

16:12

Let’s go through these stages and what is required at each stage will become clear and my request to you, as a viewer of this presentation, is first to take a moment to see where does your organization map today. Are you at stage 1, stage 2, stage 3, or stage 4? Then focus on, you know, those parts of the presentation where if you are in stage 2, be very clear that you at least do things at Stage 1 really well and understand that what you require to be at stage 3 and Stage 4. 

So, the first stage, as I said, it’s the stage where a company is doing no testing at all, and the goal for this stage is to obviously start doing testing.

17:03

And this, in my experience, is very important that at this stage where a company is not doing testing at all, even though you, as a marketer or a conversion enthusiast, might be very excited about A/B testing. I think the first thing to acknowledge is that not everyone in the organization may share your enthusiasm for A/B testing. Developers may have their reservations that you are perhaps increasing their work or your boss may have a reservation that A/B testing is just going to waste time, and obviously, the finance team may have reservations that you are spending more, and so on.

17:47

Even though no matter how much convincing you do, nothing could convince other people more than demonstrating early wins and by early wins. I mean really focusing on low hanging fruits therein. It becomes not even debatable that you made a change and that had a positive impact on signups or so on. So, I highly recommend that if not doing testing rather than going ahead and doing first really complicated tests, you focus on early wins. It is true that when [you get] early wins [by making] small changes, you will not move the needle significantly. It’s not like you’ll be able to change even the revenue bottom line of your organization just by a button change. But, you should know that your objective is not to move the needle initially. The objective is to make people understand and more open to the idea of A/B Testing. Focus on demonstrating early wins if you have not done any A/B testing at all.

18:50

And, I also recommend that you should make your first tests into a company-wide spectacle, which means that you should involve as many people as possible to gather their opinions on which variations will win. A/B testing is knowing you make an organization that there is no culture of debating in a data-driven manner whether X option is better or Y option is better. Perhaps consider making it like an internal betting game or something else so that at least even the people who are not in the marketing department understand that now the organization is going to collect data on at least some aspects of marketing. And, people pay addition to whatever you’re doing in your A/B test aspects. Once the test is complete after the initial spectacle and maybe you publish a newsletter, maybe you launch a survey with different variations, you should definitely publish learnings for both winning and failed tests to demonstrate the value of your experiment objection. 

So again, this connects to my first point that initially your idea shouldn’t be to increase the conversion rate because that happens over a period of time. Some A/B tests work, sometimes [they] don’t work, and you have very little control over that. That is why you’re doing the A/B test in the first place. You should be very clear why you’re doing A/B testing in the organization. It is to open people’s mindsets and to involve them in this idea. Right from making a spectacle to publishing learnings, just communicate, communicate, and communicate with the rest of the people in the organization. 

Bhavya: Sure. So Paras. I think this is a time when I will jump in with my question. So, in stage 1, the stage that we are talking about, I’ve also had the pleasure of speaking to a lot of customers who are stuck in stage one right now. All of them have been equally vocal about this one particular finding that I’ve got is that where do we begin? How do we start? 

Since there has never been an experimentation culture, there is a strong correlation with the fact that they’ve not gotten deep into the data. Because, if they would be playing with data, they would know what the bottlenecks are. So this also aligns with the fact that, as I said, there is no wrangling with data. There are two kinds of websites at this moment – website with low traffic and websites with high traffic. Now low traffic folks go on to methods like qualitative research. They will go and speak to their customers so that they can have an idea velocity going for them. With high traffic customers, of course, there’s been no data president as we have spoken about. In such a scenario, what do you recommend? What is the mental model that you would suggest these guys follow?

21:59

Paras: Sure. Great question. And, as I said, the reason people initially become stuck at what A/B test should I be doing this because I think they believe that they should be doing a good A/B test. And, by good A/B test, I mean an A/B test which could potentially move the needle on their organization. Perhaps, that shouldn’t be the case. Perhaps, the case should be to just start with anything that comes to your mind. I’m sure people have so many ideas that […] I don’t contest the fact that people don’t have ideas initially.

22:34

I think where people get stuck initially is they don’t know which idea to pick. My recommendation is to pick anything because what you’re trying to do initially is not to hit a home run with your first A/B test. What you’re trying to do in your first A/B test is to demonstrate the value of this practice of A/B Testing, which are two very different things. Once you’re able to do that over a period of time, as you will see in the later stages. You’ll automatically develop prioritization frameworks and different process-driven methods to get to the right A/B test amongst the sea of ideas that you have. So at stage 1, people should not worry about picking the right idea. They should pick anything that comes to their mind and run with it. Stage 2 is when in stage 1, you’ve done some A/B test to demonstrate the value of every test. You’ve made it into a spectacle to share your learnings. Now, after that, you should boost their enthusiasm. I think you’re doing this occasional A/B testing where every two weeks or so, you’re maybe running a new A/B test. 

23:49

At this stage, If A/B test is like the second priority in your organization or in the marketing teams priority list, your goals should be to transition from occasionally A/B testing to process-driven CRO. Here’s how you’ll do it.

First and which is the most obvious, you need to hire someone dedicated for testing. In the case of our customers who had tremendous success in conversion optimization. I don’t know a single instance where they were able to do that without someone dedicated for testing. By someone dedicated for testing, I mean someone whose, sort of, breathing and living testing daily. Just like if you have an SEO person or an email marketing person – they’re living and breathing the respective function. You need to recognize if you started occasionally doing A/B testing that testing is not just like a subset of any marketing activity, it’s a dedicated effort which means they should be someone who’s just living and breathing testing the entire day in your organization if you’re serious about this. To hire this person, typically I have seen, this person is sort of like a product manager. This person shouldn’t be like an engineer, and the person shouldn’t be like a marketer. The person needs to have a wider context. The person may not be very deep into a specific function. The person needs to have a wider context which typically is associated product managers to be able to see what changes could potentially increase the conversion rate for the entire marketing funnel.

25:35

And, I think, on our website vwo.com, in the resources section, you can find the job description for people whose full-time job needs to be testing.

25:48

Bhavya: Paras, does this description of this persona that you just outlaid, he has to have a wider context. I am presuming you’re also saying he needs to be a generalist? Is that what I can imply from that statement?

26:06

Paras: Yes! More or less, because this requires a variety of skills. The person needs to have an eye for good design because the person would need to coordinate with designers. The person would need to know the prima facie to see how much time coding a particular variation is going to take or whether the developers need to be involved or it can be done through vis-a-vis interface. The person needs to know where two different marketing metrics start impacting like forms and ROI of ads on Google AdWords, and so on. It requires a wide… and also say something as critical as copywriting. The person really needs to know good copywriting to write headlines, and so on. So, grammar and copy need to be good.

26:53

This is like a role meant for a generalist and over a period of time, of course, specific testing related skills become important because you get to understand CRO and they want to learn CRO a lot more. But, to start off with, you definitely want someone who is a generalist. You don’t even have to hire someone externally. I’ve seen that people who are very enthusiastic about products, they should be given a chance to transition into this role at least. This can be learned along the way. It’s not like this is some secret theory that is kept away. The only requirement is that in an eight-hour workday, the person should be dedicated to this and not necessarily bringing specialist knowledge. I don’t think that’s required currently. Once you have hired a dedicated person or promoted someone internally, my recommendation would be to put targets for testing velocity and not for conversion rate improvement.

27:59

The reason for this is that there is a very direct correlation between how many experiments you’re running and what improvement in business metrics or conversion metrics have you seen. In my experience, to put a target of conversion rate improvement [is] to say the conversion rate is 5%, and you want to move it to 7%. You really have very little control over that. What do you have control over is what activities you can do to hope that your conversion rate increases. So, put this target on the leading metric, which is testing velocity. And, this lagging metric, which is conversion improvement, will take care of it automatically. Also, by pushing people of your team to do more testing, you’re developing their mindset in coming up with a variety of ideas, prioritizing them clearly, and so on rather than just waiting for months and quarters for one perfect test that may or may not work. That’s also I have seen like it’s a rookie mistake.

29:05

Their testing is seen as an alternative for how things were done previously wherein if there was a page that needed to be revamped, the product manager will take designer in port, talk to custom – all that is important. But, if you come in where one variation is taking two months, then perhaps you’re not doing A/B correctly. Your testing roadmap needs to be full of many small tests with occasional few really big tests. For you cannot compromise on your velocity just because you want to hit a home run. 

Bhavya: Yes, that also aligns with our conversations because our customers have reported that if they’ve increased the testing velocity, there is also this additional compounded effect of learnings that come in. So, the more experiments they run, the more is their exponential increase in learning and hence, a greater probability of them coming up with a winning idea, say after 50 such ideas, the 51st would be a … there is a greater chance of it converting better or being a winner. It takes on to its own shape after a certain number of experiments have been run. So, maybe testing velocities is a bigger indicator and a leading indicator, as you said. 

Paras: Yeah, that’s a great point because testing velocity is directly correlated with the rate of learning, and you cannot really predict how much you will learn after ten tests. But, you can certainly say with confidence that you learn more after 10 tests than what you learn after one test.

30:41

Once you have put a testing velocity and put them on a calendar to create rhythm. So, every week, just like there is in the development engineering world, there is this idea of Springs and Scrums where every two weeks the development team is delivering something, and it’s like a two weekly or bi-weekly rhythm. The same thing your testing team needs to start following where without fail, every week or two weeks they’re launching something new and learning something that’s already going. You use any framework you want. 

31:24

Frankly, I’ve seen a lot of people get stuck up on the right framework. More important than having the right framework is having a framework because that will force you to start taking a few ideas every week out of the thousands of hundreds of ideas you may have. You can pick any framework you want. There are lots of frameworks available online for conversion optimization. And, vwo.com itself has an inbuilt framework that’s embedded in the product. You can choose one out of the box. But, the important thing is to have a framework and not really get stuck up on the right framework. 

So, that was stage 2. Hopefully, by then, you would have started to transition towards process-driven conversion optimization. And now your goal would be to spread, evolve from your own department of conversion optimization and testing, and spread this whatever culture you’ve developed in your small team, to the rest of the organization. Let me be very clear that perhaps moving and evolving from this stage is the toughest because it does require a buying from all sorts of facets of your organization – right from the very top, say the CEO to even departments which have nothing to do with experimentation prima facie.

If you have to embrace this culture and spread it across your entire organization, it is a tough battle, but it is doable. Hopefully, the next few tips and lessons may help you do that. Frankly, the reason you’d want to do is because not just out of your good heart, goodwill, and of course, not to underestimate that it’s important. But you should know that if other functions start embracing the culture of experimentation, your job will become very easy. Because then you’ll have fewer hurdles to cross internally and your team will be celebrated more in the entire company.

33:39

So, even for your professional career and otherwise, I think at least embracing this goal that you need other people to start thinking like experimenters too is a good goal to have. The first thing you need to do once you’ve committed yourself to spreading this beyond your testing and conversion optimization team is to should start involving top management through a formal monthly idea gathering and testing update session. I think this is important because you need to acknowledge that within your niche of testing and experimentation you may have a limited context on what’s important to business, even though you may have a lot of context of what’s important with the marketing team and the goals that are set for you by your bosses or their bosses.

34:34

But, if you go to the very top, say the C-level executives and VPs of different departments, I think you will get a much richer context of what’s important to the business. Perhaps, then you will start incorporating that richer context in your testing ideas of variations or prioritization. And after incorporating that rich context, when you feedback and see results on those ideas, you are likely to get a lot more approval in the rest of the organization as compared to if you just go back and say, “Hey, I changed the button, and it increased click-throughs.” So, you have to start embracing and incorporating the top level, higher-level agenda if you want them to start thinking about the importance of what you’re doing.

35:33

Beyond that, obviously, every month as a regular cadence this huge exposure to what you’re doing and over a period of time even if initially this seems like a formality and a forced thing, I have seen that at different levels people start opening up. Basically, A/B testing is fundamentally about ideas, and everybody likes to contribute to ideas. 

Once they start doing it, and once they start getting feedback on their own ideas, that’s when you have opened up then to start thinking about experimentation in even other decisions that are taking.

Bhavya: Yeah! Because this also corresponds with a conversation I had with an eTailer in Australia who uses VWO very frequently. He’s a power user. They said the moment they started getting executives to contribute to that discussion, their experimentation velocity went over the roof. Their velocity from 15 experiments have shot to 30 experiments just because they had these three powerful folks who knew the business inside-out, contributing to those discussions. That’s another second-order benefit coming out of it. 

Paras: Yeah, exactly! I mean if your department has been stuck because of budget constraints, you’re not able to hire a developer or not able to outsource more to an agency. If you start involving top management and they see the value they ask for a higher budget will be much easier because when they hear the results first-hand, that is a very good second-order benefit.

And, also at this stage, remembering the previous stage, you just had the one dedicated testing person and perhaps either you were relying on the design team or your developers, or you’ve outsourced. At this stage, I’ve seen most of the customers and clients of ours, VWO when they realize the importance of it and become serious. They actually commit to building an entire CRO team in-house. By the CRO team, I mean including roles from user research.

37:50

These are the people who use different analytic tools, surveys, and even do phone interviews with their customers to build customer personas. And, business analysts who are spending a lot of time digging into data funnels, cohorts, designers, and developers, and I missed it but the QA people also. Because then you are not relying on different functions and asking for favors. If you have this small team of 7-8 people, you will see the testing velocity go much, much higher. Of course, you will see the corresponding results and learning so much higher than previously. This does require investment, but without this investment, you will see small hurdles where someone is not responding for two days or the design team got a different priority from the product. All these hurdles will slow you down so much that you’ll not be able to get to the experimentation culture that you really want. 

Then again, once you start reaching out to the top management and you have the extra capacity in the CRO team to go really fast. At that stage, you need to invest in developing systems that let anybody in the organization contribute ideas, hypothesis and access results and earnings.

39:13

By this, I mean you have to start removing barriers for people outside the CRO team to interact with CRO process. Rather than relying on someone to give you an idea through email or you sharing results through newsletters, you need to automate this as much as possible so that as your CRO process runs and CRO process you prioritize, you are transparent to where ideas are coming from and who in the organization is accessing results. 

You need to become that machine which gets input from anywhere in the organization and outputs of your machine can be accessed by anyone else in the organization. That would require investment in developing such systems. Of course, nothing beats creating fun rituals. Even in stage 1, I recommended to create a ritual to gather ideas and you should double down on such rituals because, in my experience, organization’s culture changes when there is a concrete ritual associated with it, which could be part symbolic, but it could also be playing a role of actually doing positive function in the CRO process. 

These are just a few ideas you can think of. I’m sure you can think of ten more ideas. But in my experience, I have seen things like people constituting Friday of the month and calling it something like “Experiments Friday.” When that day of a month gets a name, at least it reinforces that the organization is thinking about moving to experimentation culture. Similarly, if you’ve invested in developing the system for gathering ideas, you can create an award like a quarterly or annual award for a person outside the CRO team who contributes the most testing ideas that you’re recognizing that you really care about participation from across the organization and not just CRO.

41:15

Similarly, you can create an award for recognizing a person who did an experiment in a function where no experimentation was being done before. For example, if you have a recruitment team and they are doing experiments on optimizing for candidate experience and say, testing whether people the hiring rate is better when you offer them chocolate vs. if you offer them a bag of chips or a cold drink. You can see that even though this really unusual experiment but this is happening in a context where no experimentation was being done before. Or, maybe your administration team starts A/B Testing say different lunch meals that your employees are being provided with and ratings being collected. There are so many things that can be done outside of CRO in an experimentation fashion. You need to start appreciating things. 

Bhavya: I just remember this conversation I was having with a client of ours who is a garage aggregator in the UK. What these guys did was they took CRO to a different level altogether.

It’s an extension of the Point No. 2 where one fine day they just ran out of ideas, and they invited around 50 odd garage owners who are listed on the website to come to their office and just contribute ideas for experiments for this entire one day. These guys came out at the end with a set of 30 ideas. Out of which, to date, the biggest experiment that one came from was one of those 30 and this reflects on the point that you mentioned. I mean, it doesn’t have to be just the company, it can be anybody even outside the organization who can come in. Because, that’s a view that is unbiased, unfiltered, and hence, the best view.

43:09

Paras: Yeah, I mean, that’s an amazing story and absolutely meant not just in an organization, but if you have customers, invite them. Some of the great ideas come from customers

43:26

I remember on the third point when VWO was starting out, and we were building our sales team, as an engineer who coded VWO, I was very skeptical of sales function frankly. 

I was of the opinion of why you do need salespeople to give demos, and so on. I remember what we did was to take all the free trials that we were having and only give half of them to the sales team while letting people interact with the product without the sales team for the other. We did measure what was the closing rate for salesperson vs. no-salesperson for a free trial. I was only convinced about growing the sales team after I saw that it makes a tremendous amount of difference.

44:14

And, these are small, little experiments I am talking about that VWO keeps on doing them outside of the website and product context. To build experimentation it is important these experiments and make a big deal about them.

44:32

Lastly, like I was saying, if you have built a CRO team, you can consider doing even things like putting up posters to share learning because people are less reading emails and newsletters. So, you can actually go physical in your office to spread the culture of experimentation. This is an extension of a previous point where investment into systems to contribute ideas and get results – you should also consider creating a dashboard for the entire marketing funnel that shows the number of tests running at any given moment. In an evolved culture, the entire marketing funnel is done justice because otherwise if you are not measuring which part of the funnel is running how many experiments it’s usually a few parts of the funnel where an overwhelming number of tests are being run, say top of the funnel, which is like the left side. 

But, there are other things like the middle of the funnel where they’re nurturing emails or after the funnel where customer retention things are being done like campaigns then experimentation starts getting weakened as you go below the funnel. Unless you’re really creating a graphic that shows whether your testing efforts are balanced or not balanced you may not be able to fully do justice to the entire marketing funnel and what it deserves.

And then, start celebrating inconclusive tests to prevent shaming of people who happened to have ideas that didn’t work. This is really important because if there is even a single instance of people start shaming other people because of losing tests, you will see that increasing. Over a period of time people stop giving them ideas and start filtering in the head. This is important to prevent because nobody knows initially. I mean if you knew if an idea was good, there was no point in experimentation and testing. So, you need to put good tests with inconclusive tests on the same pedestal.

46:47

You cannot say that testing and experimentation was a failure because, for one month, you had no winning result. It is a really good result that you were able to do five tests and all five of them gave winning variations. What if you created only one test and that gives you, maybe a winner or a no-winner. Again, this goes back to testing velocity. You need to focus and celebrate testing velocity rather than winners and losers. 

The last stage is there you have done this continuous experimentation across all business functions. I mean by this stage if you’ve been able to grow your organizational culture in a few quarters or years, it’s a great result that you’ve been able to derive. I think it would have been a great sort of learning experience as well. But there’s still scope for more things you can do, and I’ll just, in no particular order, talk about if you are at the very cutting edge what all can you do to improve your testing and experimentation culture.

48:00

This is beyond stage 4.

48:03

Beyond state 4 have … and all of these will seem radical ideas and some of you may even laugh. This, in my experience, is again “Beyond The Cutting Edge.” Just for the understanding which I have seen in my interactions from the people who are really thinking, say years and decades ahead. And, this is to have a cross-function experimentation function report directly to the CEO. So, the CEO can take the reporting of the sales function, marketing function, HR function [then] why shouldn’t the CEO also take the experimentation culture reporting. If you’re able to elevate this function to such a high level, obviously, you will see so many better results. This should perhaps be like a moon shot that you’re aiming for when you see where do need to take experimentation culture in your organization.

49:01

Similarly, once you have this experimentation being reported to the CEO, at that stage, experimentation is at the equal pedestal and not like a sub-function or anything else. So, you’re up to your sales team, support, the marketing team, the product team. At that stage, you should try to develop a common view and pain points of customers by sourcing data from all functions, so that you’re testing idea should not just come from like your analytics. Why can’t you tap into your support desk software to see what your customers are complaining for? Why can’t you tap into your sales conversations and see whatever the winning propositions that are being used by salespeople? You need to then expand your horizon of CRO in terms of getting data from just the website or product to any function where customer interaction happens. 

Bhavya: Does this mean Paras that the ownership of CRO is changing? I’ve had the privilege of speaking to so many customers across the globe, and this is a common recurring theme that is very visible to me and the organization here. There was a time when marketers used to own that number. And, let’s talk about four years back where they were fewer channels and marketing could have controlled all channels in terms of acquisition and yet control CRO as a function. Today, the number of channels has exploded. You have [where] e-commerce companies have to dominate YouTube. They have to dominate Snapchat. They have to dominate Instagram. They have to dominate Facebook. I mean, of course, you have your usual suspects like SEO and paid and whatnot. Thus, the focus gets lost. 

50:59

Which is where companies have now realized that you need usability experts to come in. You need web analytics folks to own that number. The purchase confidence metric is not shared. I mean, from a shared number, it’s reaching a stage where there are specialists who are focusing on that.

So is culture then pervasive? Does it have to be one team’s mantle? If not, you’ve spoken about it briefly, of course, in those slides. But, what do you think a structure should look like in years to come? A conversion structure so to speak.

Paras: I think it’s a function of where customers are choosing to interact with the company. Today, as you said, it’s no longer limited to just say website or marketing or ads, customers are interacting on social media. They’re interacting on YouTube. They’re interactive on the phone. I mean, there are so many customer touchpoints that you could; I mean you both have an opportunity to learn from customers via those touchpoints. This also includes customer behavior via those touchpoints. So, the conversion optimization, which is frankly just customer experience, making the customer experience better has to be pervasive to all those channels as well. Even though there is a growing and positive trend of CRO teams being independent, even those independent teams are still reporting either to the product function or to the marketing team.

52:36

That’s from experience. What I’m suggesting that elevating that and having that being reported to the CEO directly so that the team has a big white mandate to tap into any function in the organization and learn and influence that function directly rather than through either product or marketing. I think the rule should be that if you’re facing hurdles in optimizing a particular customer touchpoint, you need to have at least either ownership or the primary responsibility to that touchpoint. It can’t be your tertiary responsibility. Otherwise, it will not happen, and you’ll only end up optimizing just a few channels rather than all channels. Then you need to start developing personas around customer needs. The reason for you to do that is to really aim towards going from experimentation to personalization around those personas.

53:37

The reason I mentioned this here is because I’ve seen that many organizations start personalizing, or at least they buy into the story of personalization too soon. Maybe there’s a vendor who pitches a very sophisticated software or tool around personalizing and they buy into it. I’ve seen so many failures in personalization not because the tool wasn’t sufficient or they were a lack of ideas. But because the mindset was not there. Personalizing requires such an evolved mindset and such readiness of resources that in my experience, it should only happen once a company has a very good experimentation culture and a very deep understanding of customers. 

But, if you are there, the next step is definitely to look towards not just doing like very broad-based experiments, for say, all customers or all visitors. The idea should be to take the next step and start maybe running say a hundred tests for a hundred different micro personas. If not a hundred at least start with a few micro personas where you are now developing creative and variation for those micro personas rather than just generally. 

So a form should be bigger for customers who have a persona for a lot of commitment, but the same form should be shorter for the ones who are in a hurry. You need to start doing that. That’s when you will see an uplift in your business metrics. 

Bhavya: Paras when you said you’ve seen a lot of these initiatives fail and it’s not about the product it’s more about how you’ve mapped out your persona. Can you provide a more in-depth analysis of that? I mean, can you be a little more specific about the kind of failures that you have seen so that the listeners can kind of map it in their own environment and figure out where they are currently?

Paras: Sure! So personalization generally, in my experience, because it’s usually started out as tool oriented. Or, people start, sort of, developing their personalization campaigns around those tools. For example, if your tool allows for personalizing around geography, you have different content, people from a particular city or a country. That’s when you will perhaps start trying out different ideas.

56:12

What if I localize the language according to the country or what if I do maybe show local time. I mean, it could be any particular variation or creative around that particular personalization option. But maybe what your customers really require is personalization around their needs and not geography. Maybe someone from the Netherlands has exactly the same needs as someone from France. You’re just treating them differently on the assumption that because you’re testing the personalization tool gives geography options that will show different things. So, I think, the personalization effort needs to start with then introspection internally wherein you are deeply understanding customers. And, personalization only works if customers with different needs are treated differently. Different customers can be, you know, completely different. Someone could be using Linux or Mac.

57:15

They could be using mobile. They can have like thousands of attributes different. But if their need is the same, their motivation is the same, then most likely, the same sort of experiences would relate to them. You need to then personalize on those and rather than on personalization tools. And, because generally personalization starts with personalizing tools, I find that people end up concluding that it’s not having the impact that they were expecting because they are just personalizing on the wrong things. 

Bhavya: Absolutely! The other day, I was having a chat with a SaaS founder here in Gurgaon (a city in India), and he was talking about how they’ve personalized their website for geographies as you just mentioned. So they had a specific case study for APAC customers vs. US customers. Now the flip side to it that he discovered through a survey that he did, was that APAC customers actually did not want APAC case studies. They wanted the USA case studies because they were aspirational. They wanted to understand what the West is doing in terms of their own needs that they wanted to fulfill to that software. And, it just backfired. Backfired how? So these are certain things that we all take for granted whilst not identifying the subliminal needs of our audiences, and I think the groundwork should be laid in trying to understand your personas more than anything else.

58:34

Rest, everything is shallow.

Paras: Personas also not in terms of, I mean personas are approached in terms of customer attributes that X customer earns Y amount of money, lives there, does this and that. Personas should be developed around their needs and in a particular context to your product. They knew someone could be like a male or a female or earning X money or Y amount of money. But if their need is something specific and those needs align, then you need to treat them in the same bucket. 

I think perhaps either this is the last one or the second last one. Again, this is the Holy Grail, and in fact, I really see very few companies get to the stage where they have a sophisticated approach that they not just then optimize immediate conversions. They actually look across the entire funnel and optimize for the lifetime value of the customer and not just across the funnel, but even beyond the funnel in terms of repeat purchase and all that.

59:37

Thinking from the first principles of the goal of experimentation should definitely be to optimize the lifetime value of customers. This means that if you’re looking in isolation, maybe an A/B test where one A/B test has a higher drop off you might end up concluding that, that particular variation is hurting you. But if after the drop-off whatever a limited number of customers are getting into your system, they pay you a lot more as compared to the other variation. Then you have a net benefit.

1:00:11

So the only metric, that if you could keep an eye on is what are different experiments is doing to your lifetime value of the customers and not just in isolation, their respective conversion rates. And I do remember a case study on our own website where once we had changed our homepage with just a simple form. So, the entire home page was literally a form with an email address, name, and signup button for a free trial. We did see that it increased our conversion rate significantly. We had more number of free trials coming in. But, when we analyzed the data in a lot more detail, we realized that people were just signing up because we had a form in their face and not actually converting and purchasing VWO subscription later on. Overall we were actually losing out by removing friction because if we had introduced friction where people had to first read and familiarize with the product, whichever people did get in, they were of high interest as compared to the latter. So, it is important that you need to start aligning all your experiments around this lifetime value rather than immediate conversions.

1:01:30

And also, as I said, expand your horizon of who customers are. I call them non-customers, but for different functions, they are also customers. Start optimizing experiences for your vendors, employees, and potential hires. I do see at least companies like Google. They embrace this experimentation culture-making over a period of time continuously experience great for their employees, potential hires. Basically, anyone and not just customers that our company interacts with.

I think that’s it. Good luck in developing a culture of experimentation in your organization. As I said, my name is Paras Chopra, and I’m very active on Twitter. If you want to follow me, this is my handle. 

Feel free to ask me questions on Twitter which are specific to your context and organization. I’ll make sure I reply to all of those questions individually. 

Bhavya: Thank you, Paras! I had a ball. I’m sure you also would have had fun having this conversation with me. I learned a lot. 

I’m sure all our listeners are going to benefit a lot from this conversation. Before you log off, there is a question that we have. What are the books you are eating right now? What are some books you’re reading right now?

Paras: I’m actually reading a couple of books. One book I’m reading Upanishad. The Upanishad is the oldest and the most central of text in Hindu philosophy. They talk about what is knowledge, what is consciousness, and so on. I’m spending a lot of time getting familiar with Hindu philosophy. I feel like I missed that out while growing up. The second book that I’m planning to read it’s called “The Intimacies of Four Continents.” 

So, the book on how the current values of western civilization, which is liberalization, freedom, and so on. All these western liberal values, they were built upon a past which is not so liberal. It is around communalism, parallelism, and slavery. The author tries to show the conflict between the values of western civilization today and how those values were derived. For example, free trade was derived based on the ability of people to do that in the context of slavery. So, it’s a very interesting book that I’m looking forward to.

The third one I’m reading, I think it’s on a 19th-century explorer called Humboldt. I hope I’ve gotten the name right. This is a guy who Charles Darwin was inspired by. 

1:04:29

Charles Darwin actually went to his voyage and wrote the original species inspired by this guy who traveled all around the world and gave us the current modern idea of nature and our relationship with nature and how important is nature. Before that nature was seen as a tool for humans, which, to a large extent, even today, that’s the case. But, whatever we know about our ecosystems, biodiversity, and nature, we know it because of this guy. 

Bhavya: Can we expect the book summary then shortly? 

Paras: Yeah, I do that all the time on my Twitter. 

Bhavya: Have fun Paras! It was great speaking with you. I wish we can do this more often. Bye! Take care.

Paras: Okay. Thank you Bhavya! Bye.