What Is Server-Side Testing?
Over the past decade, A/B testing has evolved as one of the most popularly used strategies for growth by digital marketers, product managers, and UX designers alike. By enabling them to determine winning variations of critical webpages on their digital properties, it empowers them to optimize their experiences and in turn, their conversion funnel based on data-backed consumer insights to deliver supercharged growth.
Server-side testing is a method of A/B testing wherein the variations of a particular test are rendered directly from the webserver and then premeditatedly sent to the visitors’ device. Implementation directly on the server allows you to run more sophisticated tests that might otherwise hamper the user experience if implemented on the client-side. In addition to that, server-side testing is also opted for in cases when it is simply unfeasible to experiment on the client-side. For example, testing two different product recommendation algorithms for an eCommerce website.
Put differently; server-side testing unlocks the door to a whole new world of experimentation possibilities by allowing you to test deep within your tech stack and run advanced tests that are beyond the scope of UI or cosmetic changes. Since all the heavy lifting occurs at the server level, the user experience becomes significantly smooth and fast.
More importantly, server-side testing allows you to look beyond your website and optimize even your product, mobile app, and IoT devices’ experience. Hence, empowering product and development teams to seamlessly build truly revolutionary, experience-optimized products.
Server-Side Testing Vs. Client-Side Testing
Consider the following situation. An eCommerce store owner, with an aim to optimize for better conversions, wants to test two different checkout flows on both his/her website and mobile app at the same time. To do so, they need an advanced testing methodology, which is more inherent to the process and integrates with the website to ensure a seamless user experience. Neither of which is possible by running a simple A/B test on the client-side, and that is why server-side testing is required.
As opposed to client-side testing which involves delivering variations of a webpage that rest on the visitors’ browser (client), server-side testing allows tests to be run and modifications to be made directly on the webserver and then render the same to the visitor’s browser or mobile app. Both have varied scope and serve diverse business needs. Server-side testing is more robust and is built for fairly complex tests, while its client-side testing is meant to deal with simpler, UI-related tests.
Here’s a look at the core differences between the two:
|Parameter||Client-Side A/B Testing||Server-Side A/B Testing|
|Getting Started||Simple implementation; requires a single code||Complex implementation; additional code to be deployed to run multi-channel tests|
|Implementation of Tests||WYSIWYG/DIY editor required to create test variations||Each variation needs to be coded separately, no drag/drop available|
|Scope||Limited to cosmetic changes; experiments revolve around the design, placement, messaging of the key elements on a web page||Ability to test deeply; experiments revolve around products’ UX, algorithms’ efficiency, sign-up flows, etc|
|Performance||Asynchronous code can cause Flicker Effect|
Synchronous code can increase page load time
|No noticeable performance impact|
|Privacy||Privacy can be compromised as experiment attributes are visible on the end user’s client||More secure as experiment attributes are safe with the company servers|
|Channels||Restricted to desktop and mobile web. Doesn’t work for app.|
Cannot seamlessly run cross-channel and omnichannel tests simultaneously
|Works for both web and app.|
Can seamlessly run cross-channel and omnichannel tests simultaneously
While client-side testing can help you quickly figure out and make simpler improvements to your website that can positively impact your conversion funnel, server-side testing can help you go deeper and run advanced tests to optimize your products’ entire lifecycle. Each of the techniques complements the other and helps you overcome unique roadblocks in your optimization journey. So, deciding between the two depends completely on the scenario at hand, availability of resources, business goals, and teams’ optimization maturity.
Applications Of Server-Side Experimentation
A/B Test Deep Within Your Stack
One of the most fundamental uses of server-side testing is to test two or more variations of any webpage or mobile app. While this works just like client-side testing, the only difference is in the way variations are rendered. For server-side, it is done directly via the server while for client-side, changes are applied on the browser. A server-side testing tool is used instead of a client-side one in cases when tests are run on complex or dynamic elements that are beyond the scope of UI or when the modifications to be made are beyond the scope of a basic visual or code editor. An example of this would be running an A/B test on the shipping amount for an eCommerce store. Since the prices shown on the screen are rendered dynamically from the server, a sophisticated server-side testing tool is required to run this test.
Evaluate Your Features Before Rollout With Feature Testing
With Feature Testing, you can test multiple product features as well as multiple variations of a feature until you are sure you are rolling out the best one. Server-side testing allows you to create test variations with different product features or different variations of a product feature so you can experiment with them thoroughly, measure their impact based on critical metrics and then roll out the winning variation. Most importantly, you get real-time visibility into your products’ features’ performance. For instance, if you are a SaaS company and want to launch a free trial for your product, you can create multiple variations with different duration of free-trial periods, identify the winning variation and then make it available universally.
Manage Features’ Lifecycle With Feature Flag Management
Also referred to as Feature Toggle, Feature Flags enable you to turn your features ON/OFF and thereby alter user experience, without having to deploy new code. Once rolled out, you can simply control your features – enable them for some users, disable them for others, hide them for a certain period, and so on. If necessary, you can also roll out an incomplete feature and simply hide it until you are confident about making it live. From studying user response to a certain feature to turning it ON/OFF, all of it can be done in real-time.
Deliver Features Confidently with Staged Feature Rollouts
Testing on the server-side allows you to mitigate risk by rolling out features in stages. You can segment your audience, make a particular feature available only to a small percentage of them, measure its performance, and systematically make it available to all your customers in progressive stages (or even create your alpha and beta launch groups). You can also roll back a feature immediately in case a bug is found, modify it and re-launch it on the go. The valuable data and feedback accumulated from these iterations can help you drive constant innovation and rapidly evolve your product features.
Why Should You Use Server-Side Testing?
Build World-Class Products With Agility
By helping product managers experiment across the entire product lifecycle and architecture, server-side testing provides a giant leap towards facilitating agile product management. As a product manager, you can leverage full-stack experimentation to run countless advanced tests, figure out what’s working, generate better ideas to refine your product user experience, and deliver improved features and develop market-ready products, faster. By running advanced tests, you can better understand your product features and deliver them continuously. Not only that, with time, better experiments can help you produce more powerful and definitive results that drive fruitful decision-making.
Reduce Risk While Launching Features
Using server-side testing, you can test your features, measure their impact, make improvements and refinements to safely roll them out in stages, instead of making them live to all your users, universally, at once. This way, you can mitigate risk, learn from inconclusive tests, faster and iteratively, and as a result, launch features confidently.
Leverage Omnichannel & Cross-Platform Optimization
One of the key competencies of server-side testing is enabling omnichannel and cross-platform experimentation. Whether it be web applications, mobile apps, OTTs, beacons, or connected devices; you can experiment deeply and infinitely. You need not design tests specifically for each platform, instead, just run the same test across platforms, channels and devices, deliver consistent experiences and record the unique user behavior across all your digital properties under one roof. This way, you can gather insights on how your users respond differently to tests on different platforms, compare data for one channel to another, and use these insights to optimize your experiences across all your customer touchpoints.
Test At Scale Without Impacting Performance And Usability
One of the inevitable side effects of running a simple client-side A/B test using an asynchronous code is the Flicker Effect (also known as the Flash Of Original Content). It refers to the phenomenon wherein the original content of a page appears for a very short while before the variation loads. While there are quick fixes to minimize the time, it is almost impossible to get rid of it completely. This is harmful because the Flicker Effect not only hampers the user experience, but can also lead to a substantial contrast in the derived test results from the actual ones. Imagine if a visitor notices a particular element changing post the page has loaded. They immediately realize that they are a part of an experiment, and hence have a tendency to change their behavior on the web page.
Server-side testing, on the other hand, is immune to the Flicker Effect or any other performance-related implications; hence, you can expect accurate results while running tests on the server.
Create A Foundation For Seamless Product Delivery, Continuously
Server-side testing helps facilitate continuous integration and continuous deployment (CI/CD) in the software development lifecycle. Feature Flagging allows small pieces of code to be integrated to the main software branch and then gradually deployed for a segment of the audience, and slowly for the rest in subsequent stages. By allowing product managers and dev teams to test their features in production, make modifications iteratively, and deliver them gradually, server-side testing enables an enhanced development, delivery, and improvement speed.
When Should Server-Side Testing Be Used?
Server-side testing is most useful in the following scenarios:
Run Experiments On Single Page Applications (SPAs)
As opposed to static web sites, Single Page Applications (SPAs) serve dynamic content, which refers to modifying certain elements of a particular page based on users’ interaction with it rather than loading entirely new ones from the server. Examples of such websites include Facebook, Instagram, and all other sites/apps that work behind a login. Testing dynamic content is more technically complex and hence, can only be tackled by server-side testing tools. And in a complex website built on multiple dynamic elements, client-side testing might slow down the page load time as developers have limited control over the load time of the page rendered on the user’s browser. Moreover, since all the dynamic elements on your website are constructed and controlled by the web server, you can only flexibly test them on the server-side.
Confidently Launch New Products/Features
Before the launch of a product or a product feature, there are a plethora of ideas that product managers might want to validate to ensure that the launched product is exactly what they envisioned it to be. Server-side testing allows product and UX teams to test deeper into their stack and experiment thoroughly with their product features, measure their performance and impact, validate hypotheses, and roll products and features out confidently. The data-backed insights gathered enable them to take better decisions that help launch pioneering products, efficiently.
Run Tests Where Data Security Is Paramount
When it comes to web pages or apps that work with sensitive data, such as eCommerce payment pages and internet banking apps, etc., client-side testing can be challenging as data security is the single most uncompromisable aspect that such businesses can’t afford to be unwary of. This is the reason many marketers and product managers are hesitant towards deploying A/B tests on such pages. Server-side testing becomes the obvious choice for testing on such websites/apps as it ensures testing attributes are safe internally with the server. With server-side testing, you can experiment all you want with your critical pages without worrying about any security threat to your confidential data.
Perform A Website Revamp Or UX Overhaul
In today’s day and age, usability and user experience of your digital properties play a pivotal role in determining the fate of your business. As more and more marketers, product managers, and UX professionals are beginning to realize this, periodic website revamps or UX overhauls have become an integral part of growth roadmaps of the world’s leading businesses.
In an effort to ensure that the new version of your website or mobile app offers a significantly enhanced and seamless user experience to millions of users simultaneously, businesses need to experiment at scale. Server-side testing becomes the obvious choice as it allows you to test deeply and thoroughly to identify and fix any flaws related to site navigation, architecture, or user experience.
Enhance Mobile In-App User Experience
Mobile app technology is vastly different from web technology. So, the exact logic for client-side experimentation on websites cannot be replicated for mobile apps. Even if you do decide to implement client-side testing on your mobile app, the result will not only be poor user experience, but also tons of UI bugs. And fixing bugs on mobile apps involves a lot more steps than in the case of websites. You can’t simply ship a new version of the app and make it live to your users instantly like in the case of a website, but will have to wait for approvals from Android and iOS stores and then for users to update their apps. Therefore, to successfully run experiments on mobile apps, you will require a sophisticated server-side testing tool that can handle the complexity of a mobile app.
With server-side testing, you can run mobile in-app UX tests seamlessly without having the users download multiple variations of their app. This allows you to continually enhance your in-app user experience, roll out features that boost your conversion rates, and improve your core metrics such as customer engagement and retention.
How Does Server-Side Testing Work?
- Server-side testing requires a one-time SDK installation. The SDK is generally available in all the common backend languages such as NodeJs, PHP, Python, Java, .NET, Ruby, and platforms like iOS and Android. Once installed, SDK can be integrated into your server application by simply requiring it. This is, again, a one-time task.
- Whenever a visitor lands on the client browser, a call is sent to the backend server, where the SDK of the server-side testing tool is installed.
- The SDK acts as an interface between the website server and the testing tool’s server. It is responsible for fetching all campaign-related information from the tool’s server in JSON format, such as number of variations it has, the number of goals it contains, overall traffic distribution, and variation level traffic allocation.
- These settings are then used to create a client instance of SDK which offers various capabilities like A/B testing by providing a variation to a particular user – enabling a feature for a visitor, tracking goals, etc.
- Once the variation is assigned or the goal is tracked at server-side, the data is sent to the tool for tracking purposes. This data is fetched while viewing the campaign’s report or application dashboard.
Server-Side Testing Use Cases
• Product Recommendation Engine
To upsell, it’s almost customary for eCommerce stores to recommend a few curated and relevant products at the bottom of product pages. This product personalization for every user is based on parameters such as their browsing history, products’ popularity among other users, and so on. Now, in order to find out which of the multiple available algorithms are leading to a higher increase in sales and revenue, you need to test the efficacy of these product recommendation algorithms. Server-side testing allows you to test multiple algorithms, and determine the optimal selection of recommended products that could lead to an increase in sales and revenue.
• Cart Value Threshold For Free Shipping
The ideal cart value limit for free shipping is that sweet spot between happy and returning customers and an online store’s profitability. Since shipping fee is one of the crucial metrics in influencing customers’ purchase decisions, cautiously deciding the cart value threshold to qualify orders for free shipping becomes imperative for online retailers. From an infrastructural standpoint, testing the cart value for free shipping isn’t possible at the UI-level. You need to test on the server-side to figure out the ideal free shipping threshold that leads to increased average cart value and returning customers.
• Search Algorithm
You have a stunning catalog of endless products. But, how do you ensure that your visitors get to easily browse through it and never end up disappointed? More importantly, how do you make sure that they can quickly find what they are looking for? Whether or not visitors land where they wanted to and even consider making a purchase depends on how you showcase your products on the search results pages. This, in turn, depends on the efficacy of your search engine algorithm. Whether the products to be shown on this page should be stored basis their pricing, popularity, brand, reviews, or latest views; all can be determined by experimenting with your search algorithm. Since doing any of this will require modification to your existing code, along with the robustness and flexibility to test deep, turning to server-side testing becomes a no-brainer.
• Length Of Free-Trial and Other Website Forms
• Discounts And Deals On The Homepage
Showcasing deals and discounts on your homepage in order to entice visitors into quickly making their bookings on your website is an age-old tactic. However, factors such as the value of the discount, its duration, or eligibility criteria for a certain offer can’t be randomized. You need to test to determine the optimal values for all these variables that lead to increased conversions. Moreover, these values need to be consistent across every channel for a particular user. A robust server-side testing tool is needed to dynamically alter these values, randomly render them to the users, and ensure the same variation is sent to the user across every device or channel that they interact with.
Media & Entertainment
• Subscription Flow
Irrespective of the number of steps involved, a video streaming site’s subscription flow shouldn’t hamper the user experience. A complicated flow might lead to an increase in drop-offs. The challenge is to ensure that the flow is organic and seamless. To do so, you need to test multiple aspects of the subscription flow such as whether or not to provide social logins, the ideal number of steps involved in the checkout process, how the features of the various plans must be displayed, etc. Since each of these elements is required to interact with the entire IT system, even the slightest modification on the client-side can lead to a multitude of implications on the backend. Therefore, experimenting with the subscription flow of a video streaming platform can only be successfully handled by a server-side testing tool.
• Paywall Access
A paywall is used to restrict access to certain content on a website or app with a payment system. It limits the quantum of free content a visitor can access without a subscription. For publishers, OTT platforms, media, and entertainment websites, it is essential to optimize their paywall access to amp up their content monetization efforts. In order to do so, they need to test the ideal placement of the paywall that leads to an increased revenue while not hampering the user experience.
Testing the paywall can be done on both client and server-side. However, the client-side testing, in this case, is not foolproof as visitors can get around it and access restricted content simply by deleting their browser history or opting out of cookies. To accurately measure the impact of different variations of the paywall placement on your subscription rate, the tests will have to be run on the server-side.
• Pop-Ups In Gaming Applications
Pop-ups can make or break users’ gaming experience and greatly affect gaming apps’ conversions. Whether they are meant for in-app purchases, rewards, or hints; pop-ups should be appealing and displayed at the right stage. There are numerous server-side tests you can run to figure out which pop-up to send which player at which particular level to positively impact your core metrics. For instance, you can experiment with a certain hint by showing it to one player at level 4 and the other one at 9 to determine if this has an impact on drop-offs. The best part is that testing on the server-side allows you to do so without disrupting the users’ gaming experience.
• Mobile Banking Experience
With people exceedingly moving their financial lives online, banks have started building robust mobile banking applications that offer a multitude of services in a few clicks. However, ensuring that consumers enjoy a seamless digital banking experience can still be challenging. Whether it is the sign-up flow for a loan or the credit card payment page, there are multiple elements that can be optimized to improve user experience and make mobile banking less demanding. Since data security is utmost important and cannot be compromised in any banking application, it is recommended to experiment with them only on the server-side.
Server-Side Testing Challenges
Dependency On Developers
Right from creating and running a test to making modifications after a winning variation is declared, a dedicated development team is required to run an end-to-end server-side test. This poses two challenges. First, it makes marketers, product managers, and UXers less autonomous. They can’t single-handedly run tests and optimize user experiences, and improve experiences solely at their end. Second, there is an additional investment required for hiring resources dedicated to the same. However, if you use server-side testing continuously and strategically, and make it an inherent part of your business growth, you can rest assured of a return worth the investment.
Relatively Slower Implementation And Execution
If compared to its counterpart, server-side testing is more complex in its operation as variations have to be coded before the test can begin. Hence, implementing and executing an entire server-side test campaign can be slower than running a client-side one. However, since server-side testing is used for testing more impactful website elements and crucial product features, dedicating that extra time is immensely rewarded with deep insights into how you can optimize your entire stack, as well as UX.
Server-Side Testing Case Studies Across Industries
SaaS – Twilio
Twilio, a cloud-based communications platform, turned to deep experimentation in 2016. They wanted to experiment with their features before making them live and target them to various user segments to churn out insights that would help their developers build better products, and product teams take data-backed decisions.
They started by building the designated infrastructure required to run experiments at scale. It was comprised of a cloud data warehouse, a data visualization tool, a data analytics tool, and a dashboard to report the collected data. Next, they adopted a targeting engine that could seamlessly integrate with their tech stack and dashboard. This engine segmented users based on their behavior, targeted them with different feature variations, and recorded insights in the data warehouse. The engineers at Twilio were now able to test their high-impact feature before rollout, deliver their features confidently, and understand their customers better. Deep experimentation also helped them resolve some of their complex internal debates by validating their hypothesis.
One experiment they ran was with their sign-up flow. While one team vouched for having additional questions in the sign-up process to gain a comprehensive understanding of customers’ needs, the other argued that more number of questions could put-off the customers and potentially dissuade them from signing-up.
After a month of running the experiment, the winning variation was declared to be the one with additional questions. Contrary to popular belief, the insight the teams gained from this test was that the extra questions are not always harmful or lead to bad UX.
Travel – Travelodge
Travelodge, one of the biggest names in the hospitality industry in the UK, has always been focused on providing a great customer experience across their digital channels. However, their personalization efforts weren’t producing the expected results, and one reason concluded for the same was the heavy cost and slow speed of experimentation. They were completely dependent on their developers to re-code complex tests into their website after having identified a winning variation from a simple A/B test. As a result, they could only successfully run four tests in the entire 2017.
They needed an efficient system that enabled them to run sophisticated tests effectively and make changes live immediately by integrating changes into the existing code branch.
Switching to server-side testing not only helped them reduce costs by achieving the same outcome with a smaller development team to re-code tests into the website, but also allowed them to significantly increase the frequency and quality of their tests. It also helped them overcome issues associated with client-side A/B testing such as flickering and security concerns with tag integration. They could now run tests from their code base and make changes live instantly without negatively affecting their customer experience. As a result, they began running way more complex and deeper tests, and add value to their marketing goals. In the first year itself, they ran close to 30 tests and saw a huge increase in revenue of at least 15X ROI.
Finance – Intuit
To keep up with the fast-paced industry, Intuit, an American business and financial software company, had to ensure their products were not only the best-in-class, but were also adaptable and scalable. For their developers, this meant that they needed to roll out and test more application features, faster. To be able to achieve that, they had to ensure they had a robust system to tackle code changes during production. And for that, they realized they needed a reliable and centralized feature flagging system.
The adoption of this system enabled teams at Intuit to gain more control over their features. They could now modify functionalities by turning their features ON/OFF and target users with different feature variations. From offering a feature in beta to rolling them out to all customers; they could now control which user gets to see what and hence target them microscopically.
The system also allowed complete visibility of feature flags and provided a clear understanding of all dependencies. The teams could also control who could add or control flags internally. Also, the flags implemented would go live within seconds to ensure complete data privacy and a consistent user experience across target users almost real-time.
In the first six months, the team created more than 600 feature flags. The product and engineering teams were now more confident in delivering a new feature as they could now run multiple tests before rolling them out strategically and systematically. They could minimize risk, deliver features seamlessly, and in turn, improve their release cycles.
eCommerce – thredUP
Second-hand fashion online retailer, thredUP was always on the lookout for creative solutions to engage with their customers. During the Black Friday sale in 2016, they used server-side testing to deliver unique customer experiences using controlled feature rollouts.
From controlling which customer was being shown which promotion on the website and on email to generating and offering exclusive offer codes for each customer in real-time; they completely transformed their marketing plan for the massive sale weekend. They not only used feature flagging to switch a particular promotion code ON/OFF, but also to target various customer segments with different promotions, and deliver new content and promotions every day.
Media & Entertainment – Netflix
Netflix is a huge name in the field of testing and experimentation. The great streaming experience they provide is a result of decisions based on countless tests. One example of the same is Netflix Canada experimenting with one of their most integral features- the free trial period.
Until May 2019, Netflix had been offering a 30-day free trial to all its Canadian customers. They created four variations with 0, 7, 14, and 30 days as the free trial duration. While all the variations asked the customers for their credit card details to sign-up, the first one charged them immediately. After running the experiment for a few months, the winning variation was declared to be the first one. Since offering a free trial hadn’t helped increase the sign-ups, it was immediately done away with, and Netflix Canada began charging its customers from day one.
In today’s competitive digital landscape, customers are the ultimate source of validation your business needs (at least the only one that matters). Test deep within your entire stack and wide across your product features to let your customers decide what works and what needs to be fixed.
With server-side testing, you need not hold back with your ideas. Even the most complex of mobile app pages or most sophisticated of product features can be optimized to fuel conversions and improve your bottom line.
To ensure your website, product, and in fact, every digital touchpoint is optimized for the best, most seamless user experience, be sure to embed deep testing in your marketing and product endeavors.
With this server-side testing guide, you’re all set to get started on your deep experimentation roadmap. So, set the ball rolling, deliver delightful experiences, and build futuristic products- with server-side testing!