Server-Side Testing: Everything You Need To Know

Utkarsh: Hello, everyone. Welcome to Inside VWO, where we discuss growth ideas that you can apply to your business. Today I have Rahul Jain with me, who heads product for VWO. Welcome, Rahul!

Rahul: Hi, Utkarsh. How are you?

Utkarsh: I’m doing great. I hope you’re maintaining social distancing and taking good care of you and your family in these tough times.

Rahul: Absolutely, man. Taking all the measures possible.

Utkarsh: Yeah. So today, the topic of discussion is about how businesses are evolving in terms of experimentation. Every day, there’s a new and new business that’s coming up and they’re investing greatly in their experience. And one thing which is kind of a buzzword in the experimentation space is the Server Side Testing where businesses are not just building experiences on the client-side, on the browser where a user is opening the website but right from the server itself.

Utkarsh: So there are seamless opportunities, you know, things which businesses can do to reduce or minimize the number of hiccups that businesses get to convert their customers.

Utkarsh: So let’s talk about Server Side Testing. What is your understanding of it?

Rahul: So Utkarsh, I’ll first talk about what Server Side Testing is. So basically, you know, it is a form of experimentation where you know, whatever changes you’re making on your website are directly rendered on the web server before it is delivered to your end-user. And if I compare it to client-side testing in that scenario, all the variations are rendered on the client-side using JavaScript after the page has loaded.

Rahul: So essentially, you know, in the server-side, you are directly rendering all the changes directly from your own servers rather than, you know, a vendor’s server or on the browser itself.

Utkarsh: All right. I’m sure that you’re monitoring our own customers.. the VWO customers in terms of how they are building experiences. I see there’s an upward trend in the number of businesses that have started moving from traditional Client Side Testing to Server Side Testing. So what are some trends that you are seeing and how more and more businesses are using it?

Rahul: So, you know, before we even part of building or so it’s a testing solution. We spoke to a lot of customers and lot of prospects who showed interest in Server Side Testing. And, you know, we were able to identify three primary reasons why people want to either move to Server Side Testing or, you know, start Server-Side Testing in parallel to Client-Side Testing.

Rahul: So, you know, the number one concern that people had was performance. So, you know, when I talk about performance, I am essentially talking about, you know, the page load time. So now, more and more businesses are becoming aware of the importance of page load time and the effect it has on the business itself. So when you try to be dive on this topic, we realized that in Client-Side Testing since all the changes are applied on the browser through JavaScript, there is a certain, load that is added to the page, right.

Rahul: Just to give an example, a few years back, Amazon, you know, did a study where they found out that a hundred milliseconds of latency cost them one percent of sales.

Rahul: Similarly, Google also did similar research where they found out that an extra 0.5 seconds of delay in their search page, you know, caused a 20 percent drop in traffic, right. So, you know, at scale these are massive numbers. And these are some examples that, really tell you that why page load time and performance is a very, very important aspect.

Rahul: This was the number one reason why people just started exploring an alternate solution where, this performance could be a back, could be eliminated. And Server Side is doing exactly that. So, essentially, since you’re rendering everything directly from your own servers. So there is, you know, no JavaScript involved and there is nothing happening on the browser. So there is absolutely minimum performance impact of using any vendor.

Rahul: The second reason was that you know, a lot of companies either really learning more about experimentation they want to test out new things. They want to run more complex and advance tests. And, you know, there are certain limiting factors within Client Side Testing. So let’s take an example of an e-commerce company. So, you know, you might want to test how your search is working. Right? So, even behind search, there is an algorithm which, you know, actually tells you that hey, let’s say I’m searching for a toothpaste. Right? So I start writing a brand name. Right. And, you know, after writing a couple of letters, it starts recommending me a particular item.

Rahul: So, you know, we’ve seen companies even test out that. Whether we should start recommending after the second letter or the third letter. So running such tests, you know, which can have a big impact on your conversions is something that people are really looking forward to doing.

Rahul: Similarly, you must have seen that a lot of people like even like Amazon or any e-commerce company, they’ve done recommendation algorithms on their website. Right. And recommendation algorithms are run on, you know, different aspects. Right. For example, you know, the products that you’ve bought, in the past. Right. Or maybe they’ll recommend, you know, what is the right sort of combination. Let’s say, you know, you’re buying, let’s say, a laptop. You know, they would recommend that you buy a laptop cover along with it. So, a lot of companies, you know, are testing out, you know, what kind of recommendation algorithm works for them. So, this is, again, one of the examples of where people want to run advanced. Yes. And, you know, all these are possible with silverside testing and these are sort of limiting factor with Client-Side.

Rahul: And, the third factor, which was very common amongst, you know, all the customers that we spoke to was, you know, privacy concerns. Right. So with all the, you know, privacy-related issues that are coming up, you know, with things like ITP or maybe, even GDP or any other privacy policy that is really hampering, you know, the way all the tests are run or all the changes that are done on the browser, you know, is is a big limiting for companies. And so as a sort of eliminates that because, you know, there’s hardly anything happening on the browser. And, you know, this was the third reason where people really wanted to explore an alternate solution.

Utkarsh: Alright, it’s great that you mentioned so many use cases for Server-Side Testing. And, you know, all these use cases which you discuss in terms of, you know, whether you are looking into a search bar and if somebody – a visitor or a customer trying to search something. And you’re providing meaningful recommendations to them. All of these things are you know, I think all businesses would want to have that. But I think the problem they would run into, is in terms of understanding whether this space, this new technology is the right technology for them to invest immediately or not because I think there needs to be some maturity which needs to come from the business itself. So how would a business understand and whether they should invest in Server Side Testing or not?

Rahul: Absolutely. I think, you know, the first and the most important thing is, you know, assessing, the use case that you have, right. Where are you finding a problem? Is, you know, page load time a problem for you currently or not? You know, are you really getting affected because of, you know, the privacy-related changes that are happening these days? Also, have you reached that level where you really want to test out, you know, advanced cases? So, you know, even before, you know, considering that, you know, you want to explore such a solution, you should really assess all these points and really understand, you know, your current optimization program.

Rahul: Another thing is that you know, I feel even in terms of maturity, I think people have started exploring these tools because, you know, they want to upgrade themselves. Right. So, you know, another very important business case or for Server-Side Testing could be, you know, omnichannel experimentation. Right. So, you know, in case people are not doing that, they should realize that, hey, this has merit. Right. I should even look at it. Right. So, you know, it’s very important to look at it from both the perspective – things that you need right now and things that you should do right now. So, you know, once you have an understanding of both these things, I think that is when you can make that call whether you want, you know, also Server Side solution or not.

Utkarsh: All right. And just to deep dive into the implementation side of things, most people brand Server-Side Testing as a kind of a tech-heavy implementation or something which is usually not driven by marketers in an organization.

Utkarsh: How would you know businesses where experimentation, is primarily done through marketeers, could benefit from Server Side Testing?

Rahul: So, you know, definitely there is a prerequisite to, you know, running Server Side experimentation, you need developers who can sort of make changes to your back end because that’s how Server Side tests are run. So that’s like a prerequisite. But, you know, at the same time, if I purely talk about, you know, the need and the use cases. Right. So in that aspect, you know, both marketers or maybe product teams, you know, have a requirement for Server Side solution. Right.

Rahul: So, let’s take an example of a product team, who’s working on, let’s say, a certain part of the website, let’s say a category manager. So for them, they have they generally have a dedicated engineering team. Right. And, the problem there is that, you know, they don’t have the means to sort of, really understand their use cases and, you know, implement them in a very easy manner. And, you know, traditionally, you know, it has been like a very, very engineering-driven sort of a place where, you know, even if you’re releasing a feature or maybe you’re experimenting with a feature, it is really controlled by, developers. Right.

Rahul: So this sort of solution sort of brings up, a balance between, things that you can control. Also giving you that power to, run powerful experimentation and, releasing features with more control in your own hands.

Utkarsh: Alright. Once a business to a junction where they know that they need something like server-side kind of a thing. And I’ve heard this from quite a number of customer calls, which I’ve been to, you know, businesses can develop Server Side, in-house as well. And, you know, as long as they have a big engineering team and, you know, are willing to invest in that. So how do businesses come to this conclusion? Where they need to weigh, whether they need to build something like that in-house or whether they should look for platforms where they can subscribe to these capabilities. How do businesses come to that conclusion on their own?

Rahul: So, that’s actually a very good question. We have spoken to a lot of our customers, you know, who had like an in-house solution. But they still came to us because, you know, there are certain aspects, you know, which becomes a problem after a certain while.

Rahul: For example, there is a huge cost of ownership. Right. So, you know, whether when you’re building a solution at your own end, you need to maintain it. You need to be aware of the infrastructure and the scale that is required to handle, you know, all those experimentation changes and, you know, statistics as well. Right.

Rahul: Then, there is this important aspect of ease of use. Right. So you might build like a solution which would really help you show a certain variation to a certain audience. But, you know, you won’t have like a very easy to use interface. So that’s like another blocker. Right.

Rahul: Then one of the most important aspects is reporting. So from a lot of our customers who had a solution of their own, you know, they face a lot of problems with reporting. Like typically an in-house solution would really give you like, let’s say, a number of visitors and the conversions that are coming from a particular variation. But, you know, it won’t really give you like a statistically significant result. So, for example, most of these vendors out there provide some sort of statistics engine like Bayesian or frequentist who really power the reports and give them a significant result. So that is where you see like a big gap. And companies should really assess whether, you know, what they want, like a very, very basic solution, which is also very, very hard to maintain. Right. Or they want a solution, where dedicated people are working towards maintaining that solution, scaling that solution. You know, like, for example, we have spent years on our Bayesian algorithm. So, that is something which companies don’t have the resources and time to spend on. So I feel, you know, these are sort of very important aspects where, you know, companies should consider, whether they want to build it in-house vs go to a vendor.

Utkarsh: So you’re saying the thing that, lot of expertise which businesses, who are in this industry, have spent an enormous amount of time. So, that expertise comes into the picture when they build it. And that’s why there are some pros in terms of buying or subscribing to the solution vs building it in-house. Especially when you don’t have enough expertise in building something like that before.

Rahul: Right. And it’s also about priorities for a business. Right. I mean, building a Server Side solution might not be the biggest priority for a business. But if you get a vendor, they’d you know, they might have years of experience working with thousands of customers and they’ll constantly keep improving their product. They have dedicated people looking into that product. So, you know, all these factors also count in when you’re also thinking of building it in-house.

Utkarsh: All right. And coming to this conversation side, you know, once you’ve gone past the evaluation stage of whether to buy or build and say you decide to buy or subscribe to a platform like Server Side Testing. And, you know, and how to basically go about it, so what are the evaluating criteria?

Utkarsh: What are the evaluation criteria which businesses can use to ensure that they are choosing the best product or the best platform out there? Who can meet all their requirements? First of all, how do they prioritize their requirements? How to ensure that you know, what are those qualities and features that they should look into and evaluate while looking into the Server Side Testing platform?

Rahul: So I think, you know, the number one thing would be to really evaluate what your use cases are, right? And is the tool able to help you with those use cases? Can they solve it for you? That’s like the number one requirement. And secondly, I would say that you know what additional value are they providing. So they could be use cases that, you know, you might not have thought of. Right. Is the tool helping you upgrade yourself? Right. So that’s again, one of the important factors. Right. Thirdly, I feel, you know, robustness is very, very important. You know, they should be able to handle your traffic. They should be able to handle the scale of experimentation. Right. So infrastructure is a very, very important aspect. When you really look into, you know, the kind of tool that you were getting on board and then, you know, compliance is very, very important. So, you know, we are living in a world where, you know, security and privacy are of utmost importance. Right. And which is why, you know, it’s very important that a vendor has to be compliant in terms of, you know, their own internal processes, the way they handle your data. So all of those aspects are very, very important. Right. And lastly, I think, you know, one very important factor is, you know, support. So, you know, with a solution like Server Side Testing. Right. It might get technical at times. Right. So it’s very important to have the right support from the vendor as well. So which is a very important factor that anyone should consider before buying any product.

Utkarsh: So you mentioned about support. Right. And what kind of things which businesses would have to take care on their own if they had something like that in-house vs have a platform? I think you mentioned about support, can you unpack that in terms of what support you’re talking about here?

Rahul: So, you know, a lot of times, you know, you end up having certain challenges. So this is completely new for the industry itself, right? The way you are conducting experimentation. Right. And, you know, the chances of it going wrong can really hamper your business. So imagine that you know, you are implementing a test and all the changes are being directly rendered from the server. In that case, if something goes wrong, you know, it might, you know, impact your website directly. Right. So that means, you know, you need to be very, very careful about every experiment that you’re running the way you are running. And also, you know, the first time when you are, you know, really onboarding the tool, the implementation part of it, installing the SDK, you know, all that, you know, requires a certain help and certain best practices. Right. So in that perspective, you know, if a vendor is there to help you get properly onboard. And also, you know, help you solve all the challenges that you face while creating experiments as well. So, you know, that kind of support is very, very important.

Utkarsh: Alright. So somebody you know, who have decided to use a Server Side Platform, in-house or through a vendor, let’s brush up on the power of Server Side in terms of all things that the platform brings to the table itself. What kind of experiments you can run. Whether it’s only helping the technology team of your business or whether marketers can also, you know, get the best out of this platform, and what sort of use cases these two different types of the audience can use and run using Server Side Testing?

Rahul: So, you know, that’s a good question. So, you know, let me explain that with an example. So, let’s say there is a product team for a gaming company. Right. And, you know, it involves a product manager. It involves a bunch of developers as well. Now, traditionally, what they would do is like let’s say, you know for any product team out there. When they’re releasing a new feature. Right. Or let’s say, you know, for a gaming company, let’s say, well, you know, they want to add a new offer, you know, that will make people, you know, play more. Right. So what they would do is, you know, they would typically do like a staged rollout. We’re living in a customer-centered world, right. Where everybody’s looking for validation from the customers. Right. So typically what they’ll do is they’ll have a hypothesis in mind. You know, they would slowly roll it out. And typically, you know, companies, you know, roll it out in different phases, like, you know, beta releases, a private beta, public beta. They take feedback. The iterate upon that later. That’s like a general process today. But, you know, it is very heavily dependent on, you know, the need for developers. So, a server-side solution is really going one step ahead and giving certain control to the product manager himself. So what they can do is let’s take an example of the gaming company, for instance. You know, I know that you know, a lot of my visitors are sort of stuck at level seven right now. I might want to through a hint at level 7. So this could be an experiment for me. I would want to try out that, you know, whether it is helping people move from level 7 to level 8. Right. So now what I can do is, you know, once I sort of, you know, get this configured from my developer that, you know, hey, I want to show and offer right now, the tool will give you an easy interface for you to sort of, you know, just write down that, you know, hey, seven is a variable for me. Right. Let’s just add seven. Now, it will start showing the offer on level seven. If I type six. It will start showing me offers on level six… hints on level six. So all this control is something that, you know, product managers never had. Right. So, you know, if you want to configure it easily. Right. Similarly, if you want to run a recommendation algorithm, you know, you can simply switch. You can give a recommendation algorithm one variable, let’s say, ‘A’. A recommendation algorithm ‘2’ which is variable B. Now you can easily switch between them using the interface itself. Right. So this gives you a larger control over your releases. Right. You can decide on your own whether, you know, when you want to stop experimentation, how do you want to roll out. Right. And typically, you know, even in terms of rolling out to small audiences. Product managers used to go to, you know, engineers and used to tell them that, you know, hey, I want to roll it out to 10 percent of my audience. Right. So they would typically use like their own system to sort of configuring this. Right.

Rahul: But now, you know, in the modern systems, product managers can do that within the interface. They just need to use the scroller where they can say that, you know, hey, roll it out to 10 percent, roll it out to 10 percent more. Right. And then they also have this quick kill switch. So if they feel that, let’s say because of our new feature there are a lot of errors right now, product managers can use the kill switch. Right. So all these things are really enabling, the product teams to sort of, have more control over their releases. Right. And it is really helping them more do more experimentation in a more powerful and easier manner. Right. So this is what, I would say is like the true power of a server-side solution.

Utkarsh: All right, so what I understood that there are so many things in terms of management control over the entire product lifecycle and development cycle. And then early access and overall 100 percent rollout. So it gives overall access to all of these things as a package. And you can look into your customer journey in terms of right from you building the product and then you’re taking it to the market. And, you know, how they adopt and everything. And it gives you the power to do all of that. And the maintenance involved with it. Alright, so to conclude, I have just one question.

Utkarsh: For most people who are currently using their traditional way of testing, like, you know, which runs A/B test using Client Side Testing version? So how does me as an experimentation person, who takes up optimization for my business? How does me who’s already using the client-side variation should decide whether I’m ready or, you know, what are the indicators for me to see that, you know, whether it’s time for me to move from Client Side Testing to Server Side Testing?

Rahul: So, so honestly, I think it’s not about moving from, you know, Client Side Testing to Server Side Testing I think both should be used in conjunction. Both have certain advantages over the other. So, you know, for example, if you talk about Client Side Testing. So marketers typically have a larger controller over Client-Side Tests. You know, they have like a visual editor where they could, you know, just go in and make quick changes, you know, without needing developers. So that’s a very big advantage of, you know, using Client-Side solution.

Rahul: Secondly, you know you know, even if you talk about, like SEO impact, there is the minimum impact of an SEO on Client-Side testing because, you know, typically all the crawlers ignore changes that are done through JavaScript. Right. So, you know, overall, I would say that, you know, both of them have their own advantages. Right. So it’s not about, you know, upgrading yourself. It’s about, you know, using both of them to your maximum advantage. So if I would say that, you know, hey, I would want to quickly change a headline or maybe, you know, I would want to add a quick widget on my website, a quick pop up on my website. I would run. I would want to run, you know, a lot of growth hacks on my website. Right. So that is where, you know, you are very good with Client-Side Testing solution, you have more control. You don’t need developers. Right. So that is when you can use like a Client-Side Solution. But when you have a requirement of sort of running test, which cannot be done using Client Side Testing, like, you know, if you want to test out recommendation algorithms, search algorithms. So that is when, you know, you can use like a Client-Side solution. Right.

Rahul: Also, you know, one important aspect that we just discussed Server Side is, you know, getting more control over your releases. Right. So that is how, you know, you can use a server-side solution. But in the end, you know, I would say that both solutions complement each other. They both have their own advantages and disadvantages. And I feel that you know, both of them should be used in conjunction with each other.

Utkarsh: All right. Thanks, Rahul, especially for breaking all the myths which we had. And, you know, a lot of questions which we get through our customer interaction that, you know, people think that server-side is like the upgrade of, you know, client-side experimentation and all sorts of those things.

Utkarsh: A lot of people think about whether they should build this platform in-house or whether they need to subscribe or buy a plaotform, whatever is available in the market.Thanks for breaking those myths for us and allowing us in terms of understanding, how businesses should choose better and realize that, you know, how they can unleash the power experimentation using both of these different platforms. Thank you. And thanks for your time.

Rahul: Thank you so much, Utkarsh.

Utkarsh: Thanks