Feature Flagging Your Way to Product Success

Today, we have with us, the Associate Director of Product Management at VWO. Puneet worked with companies like Supply AI, D Shaw, and Netmeds.com before he joined us. He has 11 years of experience up his sleeves and believes that one of the most rewarding work experiences he has had was when he started his own company in 2011 which was later acquired by Knowlarity in 2014. Puneet, if you could please turn on your camera so that the audience can see you.

Thank you so much, Puneet, for doing this. I’m sure there’ll be a plethora of insights to take back from the session. Before I pass on the microphone to you, I’d like to thank everyone tuning in from across the globe and also inform you that we will be taking up questions at the end of the session. So please feel free to drop your questions at any given point during the presentation. With that, Puneet, the stage is all yours.

Puneet Kaura:

I’ll dive right into it. So feature flagging your way to product success is the title, which translates to how you can use feature flagging as a tool to achieve product success. Why we are doing this? VWO has a new product called a full stack where we allow you to use feature flagging and other related features.

It’s been in work for a couple of years, but now we are seeing an adoption curve that is growing very fast. So that is why it is a very relevant time to do this. Coming back to what we will be covering today, I have broadly divided the presentation into 4 major parts. The first one is why or how essentially product building has changed in the last decade. Coming back to the second to be able to go through an introduction of what feature flags are, the 4 Ws, who, what, when, where?

The third part will be the elements or how you choose a feature flagging tool and how it allows product managers in the jobs they do at reading. In the last, we’ll have a small demo for VWO full stack, and then we’ll later open up the stage for questions. Right? We’ll start now. So the first part is how we build products has changed dramatically.

If there is one part where I would need all your attention, it is this part because this essentially sets the stage and has a couple of good takeaways from this presentation. Before I dive into it, I would want to talk about what product success is, the philosophy, and why it matters so much. Product success definition is pretty simple. We all come from either product marketing or product manager backgrounds, where we essentially have our product areas, with customer acquisition, retention, increasing growth, and revenue.

The definition of product success is pretty clear. I want to focus on why it matters. I believe, essentially, being product people who develop products in their Internet domain, we are in a position of supreme privilege. Most of us would agree to the fact that a lot of our product releases are not actually successes. We do not get the desired outcomes, and I would pick this number broadly at 50%.

This is a luxury that a lot of other fields cannot afford. Imagine going to a doctor where you get a good treatment, 50% of the time. Imagine going to a school where only 50% of the students pass. So this is a luxury that we afford, but another asset that makes it worthwhile is what I call the power of product success. We are familiar with Pareto’s principle.

It says 20% of the efforts give you 80% of the results. This is even more skewed when it comes to web products where I would say, inevitably, 1% of products produce returns that are better than 99%. The biggest example of this philosophy that comes to my mind is a product like Amazon Prime. We all know Amazon launched Prime way back in 2008, and it has probably added a $1,000,000,000,000 market cap to Amazon Captive.

Whereas, Amazon has had very big failures, the Amazon Fire phone, which cost you $5,000,000,000. So the idea I’m trying to put across is that product success, if it turns out the way that we want it to be, it will account for a lot of failures. So achieving success is more important, than, you know, maybe, encountering a few failures. Do the parts to that success. Okay.

I want all your attention here. This is the one slide I would want you to carry as a mental model when you leave this presentation. In 2010, when I was a fresher, right after college, and started building products, the cost of being wrong was far greater than the cost of being late. And this mental model has essentially flipped in 2021. But the cost of being wrong is far, far, far more than the cost of being late.

Why has this happened? And why has this changed the way we build projects? I could narrow it down to 4 broad reasons. There is some overlap in this, the first being that the execution of products has essentially been produced. Imagine building an e-commerce store in 2010.

You had to build a shopping cart. You had to build a payment gateway. You had to build a customer communication platform, sending emails as an assist to customers. You had to do hosting on your own. Now if you look at these five parts, essentially, we have companies like Shopify, Twilio, Stripe, AWS, that have abstracted these complex parts and taken away the execution of this.

So here, the number of products that are released today is more. So you have more competition. The second part is in 2010 or the early 2010 phase your product could have 1000, 10,000, or maybe 100,000 customers, and that was considered great. You did not have the elements of social media and morality, where, again, like Pokemon Go or Fortnite can get 10,000,000 users on day 1. Also, with the rise of social media, the feedback cycles for products have gotten way faster in order of magnitude.

So you can know what people are thinking about your product, both in positive and negative terms by going to place your reviews on Twitter, and on other channels where customers talk about products. So the feedback is far quicker. 3rd point is talent density. I believe a lot of us who are attending this webinar are building our 3rd, 4th, or maybe a 5th product. So there are existing playbooks people have learned from their mistakes, and the number of people who can build good products has gone up by 500x.

The last being that we are at the end of the beginning of the era of internet access. Back in 2010, we maybe had 150 or 200,000,000 internet users. Now we have 4,000,000,000 smartphone users. So once a product hits an inflection point, it is very hard for a product with a similar philosophy to cancer. So, again, let’s say this loud and clear, right now, the cost of being late is far better than the cost of being which essentially translates if we need to achieve product success.

One single metric, that has a high probability of determining product success is the speed at which we shed. I have found this timeline again. If you are to predict the trajectory of a product being successful, you look at the shipping cadence of the deal. And more often than not, this is the greatest success syndicate. Moving on to how we can use feature flags as a tool to achieve faster kipping speed and get us closer to what we are calling product success. This is a very, monotonous or, you know, cliche slide.

I will just go through who and what part of features tagging. Feature flagging – the origins of the term are not known. It was championed by Martin Fowler, who is a distinguished technologist working at Hartworks. They’re also called feature toggles, functionalities, switches, feature flippers, and synonyms, people use different terms in different contexts. Essentially feature flags are conditional code blocks, which change their behavior based on certain metrics without developers deploying code.

And this part without deploying code is very important as this is the enabler that takes an important time-consuming part, the deployment process out of the product feedback process, which helps the product managers do their job much faster. Now the conditions for this conditional code logic can be basically your target audience. Maybe you are targeting customers in a particular country or geography. Maybe a time frame. Imagine you are releasing a certain feature for 7 days only to a certain audience.

It can be a business metric where you have this discount running only for the first 100 customers. So this is what essentially feature planning is. I would appreciate it if you could put your questions in chat. If there is something that needs more elaboration, I’ll be happy to do so. Otherwise, we can take these questions in the end.

Now where do we use feature tags? What are the typical use cases? Broadly, the feature plan use cases can be segmented into these 4 options. Canopy releases, I believe the audience already knows about this term. This is when product managers shift a feature to a very select preselected audience, maybe beta customers.

As you go to the Play Store, you have a tab to enroll in the beta. Whenever a new build comes up, you’ll be the first one to get access. So this is pretty standard. Another aspect of feature flags is the ability to roll back and do quicker releases, or what people essentially call in today’s lingo texting and production. There is a very common boxing, quote that everybody has a plan till they get punched in the face.

The product manager version of that is everybody has a sense of what the user wants to do, then the product hits the users. So, essentially feature planning is used to give users early access to a product because you have the luxury of rolling it back without involving any other members in the 3rd important use case, which VWO has been working on for the past decade and has been very successful is A/B testing or experiment. Imagine a SaaS product where you want to see if offering the 7-day trial was the 14-day trial results in better LTV of customers. Or maybe showing 2 different versions of the landing page or 2 different versions of checkout, increased checkout of conversions. The 4th part here is feature getting on operational use cases, which essentially gives the power of enabling and disabling certain features on and off to the operational heads.

Imagine the marketing manager being able to determine at what point in time should an offer run. So ideally, this was done by the dev team, and there was a collaboration between the marketing team and the dev team. But now these functions can fully reside with the operational person who is responsible for that API. Now this is the basic classification of how feature tables work. This classification is on 2 axes.

How long do these feature toggles stay in your system and how often do they change? Release toggles are the most commonly used toggles. So imagine you are releasing a feature. You do it for the first 10% of the audience. If you see that it is not affecting any other metrics, your error rate is not going up.

If any conversion metrics are not being affected to increase the release to 20%, then you release generally. So these toggles are short-lived and they change frequently. Obstggles are something I discussed in the last slide. Imagine you are a logistics manager who is responsible for controlling delivery pin codes for a particular time, maybe there’s some disturbance in this, in that area. A firsthand instance of this is when I was working for NetMeds, there was some political conundrum due to which there was a disturbance in certain pin codes, and we did not want customers to have a bad experience of ordering and then not being able to deliver in those pin codes.

And the information we were getting was not straightforward. So we had this mechanism where the logistics manager responsible for those imports could get his information from the ground, from the news, and then take a subjective call of switching the delivery on and off in batteries. Permission towers are used very seldom, and I would not allocate them. This has a great use case for future plans. Essentially, these are used to give features, you know, for authorization, maybe giving admin access or certain user access. But if this access needs to exist for a long period of time, this is not a good use case for feature flags.

The last being experimentation targets. Experiment toggles change in every request because imagine you are doing an A/B test. Every time a user comes, he might be rendered, the experiment version or the control version of the feature toggle. So they run I mean, the longevity is high and they change very frequently. So, essentially, in these two slides, I’ve captured the same essence, how and when these feature flags are used.

The 3rd aspect, I would want to discuss, and I would want to get the attendees’ opinion also here is how a product manager’s job is made easier by these teacher’s plans. So following the jobs-to-be-done framework, well, where the template is, if something happens, I should be able to do something, and then I can achieve something else. So if we follow these jobs to be done for him, there are 3 broad jobs to be done that come to my mind, which can affect or be made easier by using feature plans. I’ll go through these and then probably discuss a little more in the later half of the presentation after the demo where I would want to get your experience of how you are doing these jobs in your own companies, in your departments.

The first one is when we release software, I want to control the release process so that I can choose the timing, and audience of the release and reduce releases. Now, let’s double-click on this. We’ve all had situations where an offer needs to go live or a cohort needs to go live on a Friday evening. It creates a tense environment because now you have a couple of days off when engineers will not be available. This is a sensitive release, the product managers, and the developers are on the edge.

Because essentially as soon as the code is released, your users get access to it. So one very beautiful and important use case of feature flags is actually decoupling deployment and release. So the dev team can release once on a Friday evening or any number of times but this release will not hit the users till the product manager decides the time is right. So the deployment can happen on Friday and the product manager can start releasing the code to customers maybe on a Monday. Another good aspect of this is usually, whenever the release happens, people check for certain KPIs, and metrics, and want to make sure they are not being affected adversely.

With feature flags and pictures, the product manager has the full authority to go down the relief without having developers deploy again. Right. So the relist of the release list is essentially mitigated. Even if there is a bad release, it will only go to a certain section of users. And before anything drastic happens, it will be rolled back.

I have had instances where a bad product release has resulted in permanent damage. Be it customers reviewing the app negatively on the app store or starting to talk about it on Twitter and other channels. These kinds of scenarios can totally be a word. Phased rollout is again tied back to the second point where product managers can slowly increase the radius of release as and when they feel that the metrics are as expected. The second job to be done in which the feature plan really excels is when we experiment with features. I want to have the ability to tweak experiments without the assistance of the dev team so I can use real-time data to improve experiments. Often, the experiments that product managers launch need to be modified over a period of time.

It may be changing the target audience, it may be tweaking the traffic requirements of the experiment is really going well. You can increase the traffic allocation if there is something bad that is happening, you would want to decrease the traffic allocation. So with feature flags, the product managers have the ability to do this themselves. They are getting real-time data using the speech of platinum mechanism. They can tweak traffic, change the target audience, switch off the experiment, and all this without involving the developer.

So here’s the central theme that I’m talking about again and again – without the developer. The developer time is getting expensive. In countries like the US, an average developer hour costs around $200 to $300. So imagine If you can save this time, it actually translates into quicker product feedback, plus saving a significant amount of money for the company. The 3rd aspect or the 3rd job to be done where feature flags excel is when we release software. I want to control operational troubles without assistance from the deputy so I can work efficiently and independently.

This is purely from the perspective of ops people, not a product manager perspective. So imagine a marketing manager who wants a campaign to go live at a certain time. Usually, there will be a dev team on standby to make sure this happens precisely at the time they want. And similarly, for a logistic manager who’s trying to control some operational aspects of the business. Now with now, these people can be independently put in control of their functions and, hence, make the organization more efficient and work independently causing fewer communication errors. Coming back to the buying considerations, one philosophy I want, everyone who’s attending this to take away is even if you are not using feature flags, go back to your dev team, have a conversation about how we are handling these current use cases and how better can we do them with feature plans. If you’re already using feature flags, I would want you to go back to your stakeholders, and your developer team, and ask them how we can use them more effectively maybe if you’re using it for getting releases, how we can use it for experiments, how we can use them for operational purposes, and just help us ship product faster. As with any of those SaaS products, feature flagging also comes in 3 flavors. We have seen homegrown versions where teams realize this early and have some form of a homegrown system.

With this category becoming popular, there are a lot of open-source solutions that are coming up. And obviously, like VWO, there are other SaaS solutions that allow and totally take this pain away. And now it is somebody else’s responsibility, and you can just plug it into our system. The 4 important criteria that I would evaluate when buying the feature flag system is, first is the ability to target using the 1st party and the 3rd party data. When I say 1st party data, this is essentially the data that you are capturing.

Maybe if you’re an e-commerce system, the frequency of buyers, the average order value, the discounts you are running. So imagine you want to only target customers who are part of your loyalty tier. This needs to be there in that feature tagging system. Maybe homegrown open sources have. 3rd party data is what you are capturing via analytics, your subscription software, maybe you want to only open up certain products for people who are part of your newsletter.

So this is what I essentially mean when I say, targeting using 1st party and third-party data. 2nd important factor is, since you are using the system, it will add a small amount of overhead. This overhead needs to be negligible. There are systems like e-commerce, online travel, and betting companies where even a small, diverse effect in systems is common causing a considerable drop in conversion rates. So the idea is to keep it as minimal as possible.

The 3rd and most important criteria, see, the first two criteria are, or the first criteria is a product management criteria. The product criteria essentially target the developers who are implementing the system. Developers want the system or this external system to be very easy to integrate and maintain. And they wanna make sure of any adverse effect on the system.

Maybe if you are using a VWO feature tracking system, if there is downtime there, it doesn’t result in a bad experience for your own customers. The 4th point is a little nuanced and probably not what the great amount of discussion as of now. The feature planning system should fit together even if you might have APIs to switch on enough features. It should integrate with your project planning software and all the workflow that you have. So this is essentially broadly the criteria for determining what kind of feature clients distribution go from.

I will now run you through a small demo. So feature flagging is part of the VWO stack, which has 2 companies, which are feature rollouts and using feature facts to do experimentation. I hope everybody can see my screen. I will also show you a small glimpse of how this integration happens. I mean, how simple it is to integrate feature tagging in your code.

So imagine you have 100 customers. You already have default profiles, and now you want to roll another button in the user profile. And this is a risky proposition. It involves a change in design, and you expect that if this goes wrong, there will be significant backlash for your product. So how do you do it?

What you do is you set up a feature flagging system using VWO. So essentially this is the VWO console. These are our various products. Then I go to feature rollout. And now I have set up, the blue button feature.

This is the one that I’m looking to launch, and I want to do it iteratively in a phased rollout. I set up the feature here. So the dev team can deploy this feature independently. As I told you, one primary use of feature flagging is decoupling the deployment and the release. So the dev team, has already shipped this feature.

Now being the product manager, it is in my hands when I want to expose it to customers. So when I see here, this feature is already live in production. Right now, it is switched off and I can simply switch it on and expose only 10% of customers. So what essentially I’m trying to do here is out of these 100 customers, only 10% will get this new feature.

After these 10% of users get this feature, I would evaluate my metrics, see if everything is in order, and then increase the release rates. So I’ve already done this. Let me refresh and see. So as you can see, this release toggle has randomly picked 10% of the users, and they’ve got the new feature. Now imagine if something adverse happens, I suddenly realize this is causing a significant performance hit, and this is giving our customers a bad experience.

Without involving the developers, I can simply go to this interface and switch this feature off. Once I do that, without involving any external stakeholder, I have rolled back this feature and now users will no longer have that experience. So you see, the 10% of users that we had to roll this feature to, now they also don’t have this available. Coming back to the implementation, I understand the audience here is product managers and product marketing managers, and integrating this kind of workflow in your code essentially boils down to just three lines of code. So this is a small function that is responsible for this functionality.

What I did was, I set up the VWO system. I put in some SDK keys, and I simply ask for every user ID if this feature is enabled or not. If it is enabled, I display that code. Otherwise, I do not. So, essentially, integration is a 15-minute job.

Shanaz:

That was truly insightful. I’m sure everybody who sat through the presentation has quite a good idea of what feature flagging is and the process around that. So we have one question here. This question is by Abhishek Varma. When should we use an open-source solution versus buying a SaaS solution for feature flagging?

Puneet:

Okay. So I have strong views about this. Open source solution essentially never, but let me clarify. Only use an open-source solution at the point where it gets prohibitively expensive to use any of the SaaS solutions. I’ll give you a nuanced answer here.

See, we are all going towards a world of specialists where each product should do one thing very well. And outsource any other noncore things. Since this is an internal product, your developers will not enjoy building this out, there will be high attrition in the team that is building out this product. You will obviously not be able to achieve the kind of finesse that a SaaS provider offers. So unless you are at a point where the cost of this solution or procuring the solution is relatively expensive, which is like only 1% of the companies in the world, you should never roll out your own solution or use an open-source solution. You should always delegate this to a specialist.

Shanaz:

Alright. Thank you so much for answering that. We have another question from Troy. Troy is asking, what is the best practice for handling turning off of a feature, while users are in the system, especially ones who actively use the new feature or the workflow.

Puneet:

So, essentially, the rollback only happens when there is some adverse effect of the feature. So I don’t have a great answer to this. The pullback has to happen because it is providing a bad answer. Only in one context, this can be slightly bad. So imagine you are experimenting with something and users are exposed to that experiment, and then you want to pull back. So companies like Italy offer sticky features you can enable the option if a customer has been exposed to a certain experiment, and it’ll continue to be part of that experiment for a period of time. In any other case, where this feature is adversely affecting the performance or the customer experience, it has to be rolled back.

Shanaz:

Yeah. So there’s another question. There are two questions actually, by Kelly Twist. Kelly’s asking the first part of the question is how do you manage removing the feature flags? And the second part of the question is on the VWO form, how does the full stack link to the insights team?

Puneet:

Okay. I’ll answer the second question first. As of now, the full stack does not link to the inside tool. So the full stack product and our front-end product are separate pieces, separate containers, and they do not interact with each other. There are plans to do so, but there is no defined timeline regarding removing the feature flags.

This is a very essential element of managing the feature flag cycle. I’ve seen a lot of instances where customers do not remove these feature flags. And suddenly there is a product breakdown due to one of these. So there is no good way. Essentially, as I told you, there are 4 kinds of feature toggles.

This should be part of your clearing of technical debt to remove the dead feature plans from your system. I do not have a great process to propose here, but essentially do it after a certain period of time. Probably it should be the owners or the product manager to make sure that a feature that has been rolled out to everyone or an experiment that has ended is removed from this.

Shanaz:

The presentation was very insightful, and I’m sure everybody who attended is going to take some actionable insights. With that, I think we can close this session today. Puneet, thank you so much for doing this with us. Have a great day, everyone.