Feature rollout- How high performance teams launch features

who are also my colleagues at VWO, Sonil and Rohan. Both of them have more than a decade long experience helping companies across the globe optimize their websites and improve conversions. In the past, they’ve helped the likes of Contorian, Bain, and Hyundai in achieving conversion goals. If there’s a CRO challenge, it’s quite likely they might have already worked on it and taken it to a large conclusion.

In today’s session, they are gonna discuss a topic, which is quite unfamiliar in the CRO circle. Quite useful. Yet quite unfamiliar. That is Feature Rollout. In this session, they’re gonna talk about this capability, how can one deploy it in the respective organization, and what the use cases can cater to.

So without further ado, please welcome Rohan Shorey and Sonil Luthra.

 

Rohan:

Thank you for that introduction. Hi, everyone. I hope you guys are having a great day. My name is Rohan. I am the product training manager here at VWO. I will be taking you through the presentation for feature rollouts. And with me, I have my colleague, Sonil.

 

Sonil:

Hi, everyone. I’m Sonia Luthra. I am the Manager for Product Consultation at VWO, and I’ll walk you through the demo later.

 

Ajit:

Thank you, Rohan and Sonil. Really appreciate that you’re able to find time for this and glad to have you on board. Before I hand over the stage to you folks, I just wanted to make a small announcement to the audience. Please feel free to ask questions to Rohan and Sonil at any time during the conversation. You can send your questions via chat. Also towards the end of the webinar, we are going to ask questions directly to Rohan and Sonil.

Okay. So right now, you’re on mute mode, but towards the end of the webinar, we are going to let you unmute yourself to ask your questions directly. Okay. So with that in mind, let’s get to the webinar and also ensure that all the focus is on our wonderful speakers, Rohan and Sonil. I’m going to turn my video off and also mute myself.

So over to you, gentlemen.

 

Rohan:

Great. Alright. So, as you’re aware, we will be discussing feature rollouts today. And one of the more important aspects regarding feature rollouts is definitely feature flags. So feature flags is an engineering method that allows you to turn a selected feature on or off for your users in real-time. This helps you control your feature releases. The major criteria for this is that imagine a scenario where you have thousands of users for your website or application, and there’s a certain feature you would not like employees from companies A, D, and C to view. Well, to do that, all you need to do is enable a feature flag.

So the feature flag is basically a switch for a certain action. If the feature is true, then we can perform the said action. Now this actually comes into play, importantly, when it comes to software testing. How is that? Well, it primarily comes into play as the product filter because no matter how hard you try there will always be that tiniest bug that will creep in despite thousands of quality checks.

And we all know how that destroys the user experience. With the help of the feature flag, you actually get the ability to turn off a feature, which is, of course, a bug thereby getting you to preempt a customer experience disaster. Okay. So now, of course, we take you through what are the different contents of this particular presentation. So if we take you through the contents, you will, of course, see that we will understand how exactly product building has changed in the past decade.

We’ll be understanding feature flagging, the what, who, how, and when. You’ll also go into a certain detail, certain examples of when feature flagging should be considered, especially talking about product managers, there will, of course, be a demo towards the end by my colleague, Sonil, who will take you through a feature flagging option. So what we’ll understand first is, of course, in this, we understand the different ways that the product building has changed. As we’re all aware, there were different methodologies that came into play when we used to build products. There were different models. All of us must have come across this in the past.

And the first model that I’ll, of course, be discussing with you today is the waterfall model. Now the waterfall model is something that came into appointment before continuous deployment. Basically, organizations here used to follow this approach of the sequence of ideal steps. So what basically happened here was that the development approach was nothing but a sequence of ideal steps that one had to abide by to build great software. 

Now ideally, this is a great theory. Right? It comes into play so easily. But in practicality, it worked on the assumption that an engineer could make no mistake or that the customer requirement wouldn’t change during the course of the development.

Now we all know that’s not the real-world case. And in this scenario, implementing even a minor change meant going through the whole process again. Therefore, ensuring that changing even the width of a button could take weeks. So the whole development process worked on tickets. It was the engineering team’s responsibility to deliver the better software.

Again, you had to raise a ticket, you had to build a product, That’s not good, right, because building good products requires a lot better involvement. And this was actually handled much better with the help of continuous deployment. So continuous deployment is a methodology where engineers deploy their new code directly into the production environment, making it available to end users in real-time. So anytime a piece of code is written, it doesn’t have to wait for an enormous, delay or you know, to finally become accessible to users, there’s no waiting time in that case. Continuous deployment actually ensures that engineers the code in the production environment so that there is no delay, and it can be launched immediately during the release time.

And, of course, in this particular model, when we approach different things, you, of course, have heard of the AGILE approach. That’s, again, another very useful, very beneficial topic that a lot of us are using today with the help of those sprint cycles that make things easier. Right? 

So, of course, the other part I would like to add over here is that when we talk about the CI/CD option, that’s basically continuous integration along with continuous deployment. This is actually the agile approach, which is a lot of what is being done today in the product world. And that is basically the reverse of the waterfall model. Now this approach works on the assumption that in the world of software development, things never go according to plan, and we all know how that situation is. And it’s very important for us to follow such principles, otherwise, it could have very disastrous outcomes. So I’ll be taking you through an example just now of what a disastrous outcome can happen if the proper approaches are not followed.

So we’ve come across a very nice use case, which I hope you guys will find useful. And, I’ll just show that to you on the next slide. So here what we’ll be doing is if you are not going to be following feature flags, then you could come across a situation where you end up like my Knight capital. For those of you who are aware of this situation, great. I hope you find it interesting.

For those of you who’ve not heard of it before, I would love to explain what exactly happened here. So we’re talking about Knight Capital. This is basically a company that had nearly $440 million in assets, but it went bankrupt within 45 minutes because of a failed deployment. Okay. So what exactly happened?

Well, Knight Capital was a global financial services firm. And in 2012, it was the largest trader in US equities. The New York Stock Exchange was also planning a new program for improved pricing to investors through brokers like Knight Capital. So Knight Capital was approached for this new update. Okay.

And they, of course, went ahead with the preparations and the entire process of what they were doing, they had given it a particular code. So they were calling it SMARS. That was the core function to receive orders from the platform and then break them down into multiple smaller orders that could be matched to suitable retailers. So all the retail investors that were there, could have been approached in a better way.

Now this entire update is meant to replace old unused code that hadn’t been utilized in the past 8 years. Now the updated code had also been repurposed on an old flag, and that was earlier activated you know, to use the unused code. Now why they had kept all of the unused code for the past 8 years is still a mystery. We have no idea why something like this was required. However, what happened was that Knight Capital manually deployed that new software to a limited number of servers per day.

There were 8 servers during the deployment of the new code. However, one of the flight technicians did not copy the new code to the 8 SMARS computer servers. Okay. So SMARS is what we had understood regarding the breakup or the distribution of the shifts. Now, Knight Capital did not have a second technician review this deployment, and no one at night realized that this particular unused code had not been removed from the 8th server.

So this was a major problem. They had no idea about all of this, and there was no check. There was no thorough processing involved here. So this was a problem that has been identified. Now the real situation happened in the month of August when the minute all of this was made life.

Okay? So the minute the markets opened, like, they can be processing orders from brokers and dealers on behalf of the customers, right, based on that new program. The 7 servers that had the correct deployment began processing these orders correctly. Okay? But orders sent to the 8th server triggered, a suitable repurposed flat.

Okay. So that basically brought back that eight-year-old unused coat. Now imagine what would happen if you had a system capable of sending automated high-speed orders into the market without tracking to see if enough orders have been executed. Right? That’s chaos.

So the 8th server was activated and the functionality began routing other orders for execution. But they were not tagged properly. And this happened continuously. So it was an endless loop that was taking place. And this is actually entirely the fault of that process where they had no idea of all the unused code.

So the major takeaway over here for us is that it is not enough to build great software and test it. You also need to ensure that it is delivered to the market correctly so that your customers get the value you are delivering The engineers who deployed that code are able to work on it properly. They’re not blamed for situations like this. Right? And, of course, based on whatever happened with my capital, it shows that human error is bound to happen somewhere or the other.

And we cannot do anything in that respect because it’s a part of our life. Right? So in this thing. What do we need to rectify here, basically? Well, deployments need to be automated and repeatable as free as they can be from human potential error.

So had Knight Capital actually implemented an automated deployment system all the way completed to the configuration, deployment, and testingof the automation, this error or this nightmare could have been avoided. So I hope you found this example useful when it comes to where exactly feature flagging can be beneficial to us. Moving on to our next slide, of course, what we need to understand next is where exactly can feature flags be used So feature flags have a lot of different options available. Right? They can be used for multiple purposes.

And as you can see, feature flagging is something that’s very common nowadays. It’s basically used and championed a lot by Mister Martin Fowler. He’s written multiple blogs. You’ll find lots of documentation from him regarding the importance of feature flags. And I’ll basically try to inform you where exactly can this be beneficial to us.

So whenever we have any new feature that’s been launched, We make it a point to start small, right, because we wanna test out all the errors that are possible. We wanna test out how it’s looking on our interface. We wanna make sure that it’s not causing any disruption to our layouts. So you’ll take it a point. You’ll see that whenever Instagram and whenever any other player launches something new, they make it a point to roll it out on faces.

So we’ve all seen those new features that come up on WhatsApp or Instagram that are made available to a smaller audience. Eventually, it blows up and everyone starts using it. So that’s how they understand the potential of it. That’s how they understand how it can be maneuvered to maximum impact.

And that’s where using these feature flags, we’re able to control things in a much better manner. So these are basically the ideas that come into mind when we start feature flagging. Right? Of course, I’d like to take you through some more examples along the way. So if you want to look into other details, let’s properly look into some of the other topics that we have presented for you in the slides.

So moving on to the next topic, We understand some of the use cases for feature flags. And the first example that we’ll be going through is canary release. So like feature rollouts, can any releases are also used for testing and production? And the major aim over here is to reduce unforeseen problems and build confidence in the new release. So let’s take an example over here.

Alright? Let’s assume that there’s a client out there who is giving free delivery to all users who shop more than a certain amount on an online store. Now using networking, we deploy this new version to a small set of production machines. Alright? And in the canary phase, what happens is that most machines run once the old version of delivery charges.

Why do a smaller set of machines run on the new version? So here, we maintain a consistent experience for all the users exposed to the canary release. Right? And we make sure that they keep getting access to new features. So they’ve already made the part of that one update, and we wanna make sure that whatever new update is present, we can pass that on quite easily.

And, of course, while this can always seem to be a pro situation, certain issues might arise in such cases, we also take the help of rollbacks. Okay? So basically, if you talk about a rollback or a kill switch, whatever name you’d like to give it, this is like an undo button. Okay? We are reverting the court changes back to the previous state without even releasing that particular feature.

Right? And here, there could be certain cases where we wish to roll back the features. So one of them could be that the feature did not perform as well as we expected. That’s a high chance. The other situation is that while we were exploring this feature, we noticed that there are a lot of bugs that are needed in the production system, so we need to rectify that.

So rollback also helps us in this scenario. After that, when we talk about experimentation, now see, the ultimate idea of experimentation is that we wish to add value to products and deliver new customer experiences by bringing the best ideas. Right? So here, feature flags include powerful custom targeting functionalities, which give you access to perform real-world experiments. Alright?

And they help you understand your users in a better way. So with the feature flag, we can build quality into our product delivery by running live A/B tests on users to experiment with our ideas and finally figure out which idea works better. And finally talking about feature gating, using feature flags, we can also consider different scenarios, right, where let’s suppose we’d like to unlock certain features for users based on the service that we’re providing. So I think the most common example here would be taking the example of Netflix. They provide users with features based on the plan that you’re currently on.

So they allow different screens to be made visible or they allow different screens to have access to the platform based on the plan that you have purchased. So they have kept a check over there regarding the same. And that’s also where the feature flag can be utilized. Alright? So moving on to the next slide, You’ll see that feature flags can be used for different options.

And we categorize them into 4 different, tokens. We have released toggles. So, for example, if you have a new feature that needs to be made live, maybe for internal employees, then you can use that as a release toggle. You can use an operational toggle when there’s a certain feature that is readily available, but you wanna make it advanced in two different environments. You can also do that.

You have permission toggles that can be made live only based on certain prerequisites. So if someone is a high-paying customer, you’d like to give them extra permissions. You can do that. You also have the experimentation toggles, which we already took into consideration that you wanted to make certain changes live. Based on the different tests that you wish to accomplish. So that’s where the experimentation tokens come in. And after this, we’ll, of course, also show you how we can launch all of these different feature flags and what are the different scenarios that can come up. So I’m excited to show you the next slide which has a proper document, you know, a proper video. Oh, sorry, a proper image for you to give where you can see that now we’re moving beyond that step where we just implement changes on buttons or text. Right?

We’re moving to this next layout where we’re also going to be able to modify templates. We’re able to modify layouts, pricing, and a lot of back-end activities as well. I can control them in a better way and understand which one is the most suitable response or the most suitable variation that can be displayed to all our visitors. And from that, of course, we can declare a proper window.

So here, if you take a look at the next image too, you’ll be able to see that in the next image, we have also covered how you can now run your beta programs more efficiently. Because feature flags act as toggles. So you can make it live for a smaller audience at the beginning. You can see how they respond to that particular new program and accordingly build forward from there. 

Take examples of how rings and how all of these were slowly launched. Of course, now when we deliver features, we can make it more standardized. So you can start small within your team. You can move forward to the beta version you can then make staged rollouts for different types of customers that you have. And finally, when you’re completely ready, you can make it available for every single visitor that comes to your website or application. 

So that is the power and the potential of feature flags. Moving forward to our next slide, when we take a look, we’ll also see how this comes into play when you wish to buy, why exactly would it be better for you to purchase this rather than build it on your own?

So there are multiple instances where this comes into play, and basically, it is the idea of buy versus built that comes into play here. Okay? So here, if you were to build everything from scratch, I think it would be a very tiresome, very long process where you’d have to make sure how it works for different environments. You have to constantly be in touch with your development team. Google focuses primarily on this aspect. That you have to get something ready to build something in this case. Whereas if you purchase a product that serves all these needs, you can manage things in a better way because they will be catering to your service most suitably. So here, it makes it easier for your developers and your colleagues to be able to maintain things. 

Your product managers are more well-versed they know that they can reach out and, talk to the respective service providers in a much better way, and they can get things done. So in this process of, you know, buying versus building open source versus homegrown, you can see that it helps if you take the help of certain experts in utilizing these features rather than building them again from scratch, and with that, of course, I will be passing on, this mic to my colleague, Sonil, who will be taking you through a demo of the VWO platform. So thank you so much for your time today. And, yeah, Sunil, it’s all back to you.

 

Sonil:

Hey, everyone. I’ll walk you to the VWO demo for the feature flags And, before we jump in, I’m just gonna show you the VWO platform. And here’s the first look at the VWO FullStack dashboard here, and, this is how we start with projects. Projects are more like workspaces where we define, like, what percentage, like, where we can define environments and even associate campaigns into it. For example, we support multiple languages out there.

We support like Node JS, Python, PHP, and everything. Each environment gets you an SDK key that is related to that sort of environment, and you can run tests or routes specifically for that environment. You can switch from your state to production. Now, of course, you can be very easy. 

A quick high-level overview of how VWO FullStack, which also includes how the VWO Feature Rollouts work. We give out an SDK that gets hosted in your backend system, and then in your back-end service, we use API calls from your server itself where the SDKs host. Now there’s no need for third-party API calls going outside of VWO. Hence, this makes VWO the fastest product out there in the market. If you talk about latency, reaching out to your server will always be 0 ms and nothing is faster than 0 ms. Now once we have reached out to the VWS SDK, it gives out a response that should bucket this user in this rollout or not, depending on the rollout percentage. Now it depends on the feature section here. The feature section is like a set of variables.

We start with defining some variables here. And based on these variables, once they have been defined, we can roll out them to the users. These are the values that we have defined for the variable. It can be tied up to any feature, it can be a bullion flag, integer, a string, or even a set on JSON.

We support all types here. Once this has been defined, we’ll move on to the feature rollout. The setup is just a single setup. We select the feature that we want to roll it out. Define a campaign key here. Once the campaign key is defined, I’ll select the project. Each project has environments, and a list of environments will be populated here. Let’s say I’ll start with the UAT first. In my UAT environment, I want to run it only for 50% of my users. This can be my internal error management.

Then No doubt to 100%. I can add my production environment here as well. Turn off my UAT enabled for 20%. I will see those features and I can experience those features. I can easily increase this gradually depending on how things are working out. Maybe let’s say things are going there. Like, people think of my feature, and they are interacting with it. I can keep on increasing it. 

Let’s say after a week or so, things go south and you feel that roll it back on all fields. You don’t need to do anything in your mapping. It’s just a toggle switch here. Once you click, the feature has been turned off for everyone. Now the magic lies in the code section here. There may be questions about how is VWO updating those variables.

We use the code here, which is like the first feature-enabled API. This buckets the user and says whether this feature should be enabled for this user or not. If the decision has been given out, we will show them the VWO feature that we have defined, and the value of these variables will be updated by the SDK. The SDK has a settings file that has all the conflicts, like which variable should have and what value. We ask for some parameters here, which are user ID and options. User ID is like a primary key, and options are more like custom variables or user attributes. You can use those user attributes here, for segmentation. 

Let’s say there is some demographic information, or maybe let’s say there’s a type of user that you have and that is a premium user. Only premium users should be part of this rollout. I can define it here. I have an option for adding multiple variables with admin or separators.

Even logic using brackets too. You can save the segment, reuse it in your other feature rollouts, or test it as well. We also offer you a forceful bucketing if required. For example, only your internal users should be part of this. You can define the user IDs here with a comma separator or you can upload a small list of 100 to 100 users and then run that feature a lot only to that list of users. You gotta schedule an option here. Let’s say you want to schedule the start or stop of this feature or allow it at a certain date and time, you can do it. We can also do it to tell the next number of visitors. 

For example, only the first 10,000 users should be part of this test. I can easily schedule it that way. So that’s how we set up a VWO feature rollout. The technical documentation behind how we set this up is available and hosted at VWO Dev docs here. We will share this with you, and I’m dropping this URL in the chat window as well for you guys to have quick access to it. So once this is set up, I have this feature created, and I can easily make it live.

By just enabling it from the VWO dashboard. You can just hit start now, and the feature may go live with the desired percentage that you have set up. We also have an option to start later. You can keep it in drafts, and that can be used later. 

The report will show you the number of users tagged here with the number of, like, how many users depending on your traffic have been part of this rollout. That can be downloaded as well. VWO also offers you a feature test and an A/B test. Feature test is where you can define and verify, you know, like, run multiple tests at the same time and test out different values of all the variables that you have created as a feature test, or you can test out different algorithms recommendation hinges using VWO A/B testing as well. 

Over to you, Ajit.

 

Ajit:

Thank you folks for this beautiful session. Enjoyed it. And I hope our audience enjoyed it too, and they emerged from the session, more intelligent than they went in. I see that we have got a question from Daniel in the chat box. Also, I would encourage everybody else to ask questions to Sonil and Rohan.

So we have got a question from Daniel, and he has asked, hi, Rohan. One second. Hi, Rohan. Can you give me an example of how an e-commerce company uses a feature rollout in the organization? So, yeah, that’s the question.

Was it was it understandable, Rohan? Can you please help?

 

Rohan:

Hi, Daniel. Thank you for that question. Of course. I think, primarily, you want to understand how exactly can we launch those features using you know, the feature role adoption. So it’s like stage rollouts that are used.

Alright? For example, let’s say there’s an e-commerce store that has a big sale that they’re planning in the coming months. And they wanna try out a situation where maybe to a certain segment of users, they provide free delivery up to, let’s say, 2 miles than to a different segment of users. They like to provide delivery at half cost. Okay.

But they’re also reducing the prices. Of their products to a certain extent there. So whenever there are variable natures involved, right when it comes to dynamic pricing, when it comes to overhead costs, they can play around with these values and accordingly figure out which one works best for their users. So are they more inclined to purchase if there are, lower delivery charges, or are they more inclined to purchase? Maybe if they’re they’re offering free one-day delivery, or maybe one-day delivery with an extra overhead cost. So that’s where these staged rollouts can come into play. I hope that answers your question.

 

Ajit:

Yeah. Do we have another question? Okay. That’s all the questions that we have at the moment. Folks, I’ll give you another minute to ask, meanwhile, let’s see.

We have got another from Martha. Martha asked about mentioning our feature toggle and the feature flags follow the same thing.

 

Rohan:

Primarily, they are the same thing. So they call feature flags. They are called feature toggles. You can call them either way. And it’s based the reason behind that is like, you know, it’s like those switches that you get at home. They turn the light on and off. So it’s that easy. So you’re basically either clicking on the on option or the off option like my colleagues have shown that you can easily switch it on and off. That’s why it’s called the beach at toggling option as well.

 

Ajit:

Yeah. So that’s all the question that we have. At the moment, folks, you are encouraged to ask questions to, Rohan and Sonil on the call as well. You can reach out to them at Rohan and Sonil, can you please drop your email IDs in the chat, or maybe We do lose email IDs in the chat so that we can discuss them later on should they have one? So please, share your questions or whatever your feedback for the session on this email ID.

I will also be sending a post-webinar email to all of you, whether you’re attending or not, I’ll be sending an email to all of them. Sharing the recording of the webinar as well as the slides. So you can respond to that email as well. So till then, thank you so much for attending. Thank you, everyone, for this beautiful session.

I hope our audience enjoyed it. I enjoyed it a lot. And, again, thank you so much for taking the time to do this. So, thank you so much. And that’s all included in the session. Folks don’t miss to ask questions.

 

Rohan:

Alright. Have a great day. Bye.