Product Management and Experimentation

[00:00:00] Vipul Bansal: Hello, everyone. Welcome to ConvEx ’23. VWO’s annual virtual summit for growth and experimentation professionals. My name is Vipul and I’m a Senior Marketing Manager at VWO. 

[00:00:19] Thousands of brands across the globe use VWO as their experimentation platform to run AB tests on their websites, apps, and products.

[00:00:27] I’m excited to have Pius with me here, who is a product manager at Canva. Hey Pius, excited to speak with you. 

[00:00:35] Pius Binder: Hi, nice to meet you. I’m Pius from Canva and I’m a product manager there working on user platform. 

[00:00:43] Vipul Bansal: Awesome. Great. So, Pius, can you share more about your role as a product leader for Canva and, how does it contribute to the success of the company?

[00:00:55] Pius Binder: There are many different people that contribute to the success of Canva, not just product managers, there are many leaders and people who drive that. For my particular role, I’m currently working in the user platform team at Canva, and what it means is that all the data comes together of users and we are building a platform to [make] Canva be successful.

[00:01:19] Originally, I worked on a lot of experimentation and different things at Facebook as well, and for my own business and Canva. And, now I’m working on building out the platform to help with the future growth. 

[00:01:32] Vipul Bansal: You manage two teams at Canva. And of course, I would love to know what that is. But I would also love to know, how do you manage and prioritize managing these two teams? And what strategies have you found to be effective?

[00:01:47] I believe this question would be really helpful for the segment of the audience who are new managers and they’re looking for strategies to manage people. 

[00:01:58] Pius Binder: I mean usually it’s not a choice. You start like with one problem and then it becomes two and it becomes 10 opportunities.

[00:02:08] And the teams usually, or the people usually follow the priorities of the business. And this is also what happened here. So there was just a need that emerged in the business and then it became bigger, and we needed more focus. So that’s how two teams emerged. I cannot lie about it, but having a strong tech partner is always a good thing. Tech lead. 

[00:02:38] Because a lot of the things in software development depend on the tech partner. And I am very lucky in user platform to have a very strong tech lead. I was actually always lucky in Canva, like all the engineers are really awesome. However, there are really a lot of different people that come together in the team to have ideas and bring things forward.

[00:03:06] And then there are other teams that you need to collaborate or [form] groups. And so it’s like a village or city of different villages where everything wants a little bit. So to me, the most important success factor is the people and making sure that everyone is on the same page when it comes to priorities and also be like friends or friendly with each other.

[00:03:31] It’s the biggest problems I’ve seen when people are not collaborating. And then it makes things very hard. Because a lot of the decisions in bigger communities are not logical. It’s usually always about emotion. Someone wants something and achieve something. I think with all emotions, if we were all robots, it was quite simple, but it’s not the case because we are building products for humans.

[00:03:58] Therefore, humans also need to decide. And there are also emotions involved and it’s important to manage those emotions of the people well in order to move things forward quickly and have high impact. And that’s the same for producing work in two teams. So you need to have the vision, the direction, and then also be very patient in keep moving things forward.

[00:04:21] Vipul Bansal: You also mentioned about building a great authentication experience. And as a user myself of different digital products, of course, that is a really key thing for me as well. So, first of all, can you discuss the importance, from your perspective, of building a great user authentication experience?

[00:04:40] And then, of course, how have you helped improve this experience at Canva? 

[00:04:47] Pius Binder: I cannot speak for Canva particularly but there are a few parts of the product where friction is always an issue. And that’s authentication and payment. Like, very simplified.

[00:05:06] Maybe there’s more. Like key actions users need to take. A lot of the value and the need of the product is created somewhere else. So, it’s like the editor, for example in Canva, or for transfer bias, it is the transfer or there are like different things that are important for different products that create the product-market fit.

[00:05:27] For authentication, it is very important for Canva to be frictionless. And easy. And people forget their emails and they login with different emails. And the oldest problems that you probably all know as consumers using your own apps like Netflix and this type of stuff. 

[00:05:47] So from my personal perspective for authentication, it’s a very clear thing, which is making it very easy for people to access the product, as frictionless as possible. And access means either you’re an existing users and coming back, or you are new users. 

[00:06:07] Vipul Bansal: So, the next thing which comes just after the authentication experience is the first impression of the product, as soon as you enter into the product beyond the login window.

[00:06:18] As a product manager you have that responsibility to build a great positive first experience. What steps have you taken to build that great first experience for new Canva users? And also, how do you measure the success of those efforts?

[00:06:38] Pius Binder: I cannot speak again for Canva, but I can speak [in] generic [terms]. First of all, everyone in product contributes to the first experience. Because there’s so many different scenarios and how we can access a product, not just in Canva, but also how we record this session or how people view this experience that we are creating.

[00:07:05] So everyone plays a part of that, from marketing to CHT. And I was playing only a small part in a bigger effort. And the bigger effort is really everyone needs to make sure that the first experience [of the] product is great.

[00:07:25] I think measuring the first experience is incredible. Text needs to be incredible and also changes as the product evolves. Looking back at the history of Canva from my own perspective. In the beginning, it was used for different purpose than it is now. Like now they’re more doc types. You can create presentations and documents.

[00:07:51] And in the past, some of it wasn’t possible. So it’s very important to cater to the different needs of new personas emerging in a product. And to do that well, [it requires] a collaboration effort and making sure that the user experience doesn’t break. In terms of measurement, I’m wearing now my product growth hat. 

[00:08:17] To me it’s very simple. It’s like I have the best activation possible in the fastest time and then this leads to more retention. That’s the simplified metric answer.

[00:08:28] But then when you think about it, what does activation mean? And activation is very different for every product. And I think that it is important to align first with the business and the stakeholders or with the place you work in, on what the thing first experience is, and what the aha moment or the critical moment is for people to join a product. And then setting this as a metric to activate. And then you can put all the efforts into measuring activations and hopefully don’t change it as much as you have like different initiatives that improve it.

[00:09:14] Vipul Bansal: My next question is also related to some of your work at Canva. But, of course, hiding all the sensitive information because you mentioned on [your] LinkedIn profile about growing presentation users by X percentage. Without revealing that X, if you can just let us know, what did you do to increase the presentation users, it’ll be great. 

[00:09:40] Pius Binder: The presentations use case is an interesting one. It’s about verticalization of a product and I used to do that in Facebook as well where I helped verticalize new products. I’ve worked on Facebook pages [for] pages growth, and then the product was radicalized with marketplaces, offers, cars, different new products or verticals part of the Facebook page experience. And Canva presentation wasn’t that different. Because Canva presentation was a new vertical within Canva. So, finding the appropriate moments to cross sell of a product that already has product-market fit is important.

[00:10:27] So for presentation videos, there was already product-market fit. So most of the work is already done. So there’s like a lot of people already thinking very freaking hard on how to create a great product. And then I basically help to find the right moments and channels to cross sell the product in the right moment.

[00:10:53] And, it is usually always the same thing. You can extend the channels, so you can move from different channels that are not there. For example, email isn’t there. You can add it like similar to Uber promoting Uber Eats, making it one of the fastest growing products in Uber, or you can add notifications that have not been there yet.

[00:11:18] So you can extend the channel. You can find new behavioral triggers. You think people should create presentations, but they are not and help them to discover the product. And that’s mainly around product optimization. Things that I haven’t covered, but that are also incredibly important is, for example, activation with marketing.

[00:11:45] But that’s really something I haven’t done. But that’s important as well that you actually broaden the audience and the target market for an existing product or new vertical. 

[00:11:57] Vipul Bansal: Got it. So with all these features that that you’re working on and launching on a consistent basis, are there any experiments that you’re running?

[00:12:06] And, and it’ll be great if you can also share a few examples of the tests that you’ve run. Oh, of course, feel free to hide any sensitive.

[00:12:14] Pius Binder: Yeah. I think a big part of not Canva but for any optimization for growth is, and that’s a good question is, helping the user to make fewer decisions.

[00:12:33] So when you work in a company that creates a product, you really understand the customer well, you really understand the product well. But there is a big bias on interviewing the same customers qualitatively and which are power users already. Or you see the product in a way that most people don’t.

[00:12:59] And this is where data informed decision making is incredibly helpful. Because you cannot just use data to make decisions. It’s important to inform your decisions with it. And one of the things that always work in any circumstance for any [company], [be it] Facebook, or my own business, [or] Canva, and also when I speak to other startups I help is just having fewer actions. 

[00:13:31] Like very often they’re like too many call to action buttons. And the reason is that there’s maybe disagreement between the company on what to show and it always comes at the dis-benefit for the user and also for the metric. Usually it can be revenue or active users because there’s a indecisiveness on what action should people take and then instead of one decision to just adds two buttons there. And that’s a lot of the work that I know that is always helpful like reducing the number of actions for users to take. And that’s all very powerful.

[00:14:10] Vipul Bansal: I was also curious to know that with all this experimentation, of course, experimentation on features is something which I have heard from a lot of companies that they do. But is there also a way to identify your ICPs, your ideal customer profile through experimentation? 

[00:14:31] Pius Binder: I don’t know if you can find it with experimentation, the ideal customer profile. I’m not sure I need to think about it. But the experimentation or AB testing helps you [to find out] if experimentation is for me. You have a control in the test group.

[00:14:48] That’s for me an experiment. But what you can know is that you can figure out what makes people more bought into a product and creates more power users. If this is a ideal customer profile. So you can help how you can create more frequent users using the product, for example.

[00:15:18] It really depends on the use case of the product to define these activities. I think that your ideal customer profile, you can only find it with qualitative research mainly and some quantitative. Because you really need to know what you want first and you cannot find this with experimentation. Because experimentation is just a guide for decision making in my mind and it’s not so much about finding the ideal customer. Because the ideal customer is maybe some somebody who doesn’t exist yet. 

[00:15:53] Vipul Bansal: You are surrounded by a plethora of metrics. It’s difficult and also not very relevant to track just every metric that you see in front of you. Have you identified any meaningful metrics to measure success? And how do you ensure that your teams are aligned and focused on these metrics? 

[00:16:15] Pius Binder: It’s a good question. Generally, over time, I ran around 1,000 experiments so far at Facebook, Canva, and my own company.

[00:16:29] So I have a lot of exposure to experimental design. Like basically setting up this little sign tests and, making decisions based on that. So the red thread for all of this is that less is more. It’s very easy to add a lot of different metrics there, but then, even if you align before on what to measure, the communication around it becomes incredibly hard.

[00:16:58] So if you have a lot of different things to measure, it becomes incredibly hard to communicate. Because one metric goes up, one goes down, and so on. So first, I think it’s important to agree, as I mentioned before, what do you want to measure and why, and agree on one metric, ideally maximum two, to help make the decision making later. Because if it’s too many, what does the primary metric mean? There’s only one primary metric. It’s not that’s why it’s called primary or one. And really rank the metrics you want to measure. Second, it’s very important to have an end state defined before the experiment. Because if you say, “Oh, let’s just wait until the results of those perhaps it’s not the right experiment to run”. 

[00:17:47] So it’s okay to call it off and rethink about something because also neutral results are learnings. But I see that a lot [of people] that are keeping the experiment running a little bit longer to see whether the data is going to become statistical significant.

[00:18:02] And that’s not really always helpful on faster decision making. Because during the time you wait, you cannot make a decision. You just like block. And also you train or you work with the team to start creating their own hypotheses and own experiments to empower them to come up with ideas and things to test.

[00:18:22] That’s important because that can be a big lever for a product to grow quickly without you needing to come up with ideas constantly. So I think this is also a very good tactic and way of doing it. 

[00:18:38] Vipul Bansal: If you can just name a few metrics that that you track as a product manager.

[00:18:43] Pius Binder: Usual metrics are like, active users for one day. I think the one day active user metrics are good. Because you can have like 28 days or seven days, but then it takes much longer having to experiment life. And with a lot of things you should see change very early on. If not, just having longer 7 day active users, or the 14 day active users metrics, doesn’t help you to get better insights. Then you want to have revenue metrics like either trials or payments completed. And if you cannot measure active user metrics, you want to have like sessions or something like that. Like some other proxy to know how active people are.

[00:19:36] Vipul Bansal: That’s great. Thank you so much for sharing those metrics on our request, Pius. Quickly progressing with the discussion, a question which has a very subjective answer. Of course, depends person to person. But I also wanted to know from you rather is, what do you see as a future of experimentation in product management?

[00:20:00] Of course, it might sound really a vague question. Everybody can make out their own sense of it. But as a product manager yourself and doing experimentation, [how] do you believe this field evolves in the future? 

[00:20:16] Pius Binder: Yeah, that’s a interesting question. I think about it a lot. What will happen and, what happened in the last couple of years and where it’s going?

[00:20:30] I think first of all, all this experimentation stuff is much more accessible. So you have initiatives from you, where are your company, where you try to educate people around experimentation, there are more tools. There is your tool (VWO), and Google Optimize. It’s easy to run experiments. So I think it’s generally like more accessible running experiments with different tools.

[00:20:55] And then there’s more content out there. So I think the internet is much richer with information around that. Which I think is great. 

[00:21:04] But one thing hasn’t changed, which is data literacy. So I think data literacy is something that [is same] over the last eight years.

[00:21:18] Is it statsig? People don’t know what statistical significance means, first of all, but they ask for it. How big is the impact? How do we know this is right? And then there are a lot of biases and misconcepts that are always the same. And depending on the companies, like in some company, there are more technical people. In some there are more designers or non-technical people. 

[00:21:47] And there are different philosophies on how to drive the company forward. For example, Snapchat and Facebook. Snapchat became very successful, next to Facebook, with using a design first approach. So the CEO is a designer. And somehow experimentation is everything at Facebook. AB testing, it’s the core, the DNA of the company. And still Snapchat had a lot of traction. So somehow there are other ways on achieving incredible growth with not too much experimentation. So how can it be? So I think there is value in either approach. But, eventually Snapchat, I believe had to pivot and be like more data driven as well.

[00:22:29] That’s what I’ve heard. So, they had this ceiling point. And at some point companies need to use more data to make decisions. And it will happen sooner or later. There’s no way around it. But the discussions I’m hearing and the feedback has been, “Oh, how do we know it’s not harming the users? Like, is it really more active?”

[00:22:50] There are still a lot of conversations around that are being held. And I think that’s a big problem to solve. Like how to adequate people and what this data means. The other parts that I see emerging is like the usage of dark patterns. I wasn’t designing things to trick users.

[00:23:13] I think in 10 years ago, it was more acceptable on the internet to have like, make it a bit harder for people to unsubscribe, or make it a bit harder for people to cancel the paid subscription. I think there’s like a very clear trend that this is stopping. And I believe there was some regulations in place in some countries, and that’s something just to consider.

[00:23:36] You cannot assign stuff anymore that is harmful for people. And that changed on how the internet evolves and people think about it and the responsibilities of companies. And the last part is GDPR and data compliance, and how to treat the data. That’s also like a big topic. And last on the opportunity side, I think we speak a lot about AI and machine learning in experimentation.

[00:24:08] But I haven’t seen really besides in some very particular products, like for example, the news feeds of Facebook or in TikTok, the feeds that makes you also addicted to see the next reel. And I really haven’t seen this vision on totally automated experimentation and AB testing.

[00:24:30] I think this is still something where the human, I believe the human decision making is very, very important to curate what is shown to users. So in email you can do all types of stuff to automate and show people different things. And sometimes the results are not that much greater with more technology.

[00:24:51] That’s, I find very interesting because it means that AI is there and we use AI to define segments of users and be smart in the targeting. But people are used to that on the internet. So maybe, this is something where I feel the creator of the human of the product manager is very, very important.

[00:25:11] It’s going to be important on helping to make the decisions on what to bring forward into the product. 

[00:25:17] Vipul Bansal: Great to hear from you a peek into the future, I would say.

[00:25:23] And we sort of both agree that of course, we can’t completely rely on AI yet. We still need to have a human to put their brain into the entire process. Only then I would say, we will match the expected quality of work. Thank you so much for sharing those pointers, Pius.

[00:25:48] This brings us to the end of our discussion.

[00:25:51] I love to ask this question to almost everyone that I speak, what books are you currently reading? And if you’re watching any web series, it’d be great to get some recommendations there as well.

[00:26:06] Pius Binder: I used to read a lot of books about psychology and habits and stuff. Like the power of habits to understand the psychology. One book I think that was really great was Growth Hacking. That’s experimentation one-on-one on how to establish the identity of growth and experimentation. And the author also describes very well on where to start. Because sometimes in which part of the product should you start experimentation. And the author has a very clear view that it should be the paycheck of experience makes sense because he can add a lot of value quickly for the business and the customer.

[00:26:57] So, I love this book and I used a lot of this in my work at it’s for another company. I’m reading also a lot of books right now around understanding humans differently. So about the philosophy of life. I read the The Tourist from Auschwitz. It’s like a very dark book but with a positive outlook on what our role of humans should be in society.

[00:27:35] And I believe, it is important because in technology we are creating a lot of wealth. Or we contribute to a shift of wealth of societies. And we can all agree the society is currently moving, maybe in a direction where we don’t know if the society is going to be more equal or less equal.

[00:27:57] And, 

[00:27:58] That’s one of the philosophical book. I highly recommend these books as well.

[00:28:02] And then the last book I read was about negotiation and the name of the book was, [Never Split the Difference:] Negotiating As If Your Life Depended On It. I think this book just revealed, especially for working companies a lot of tools and tricks on how to ask questions in a different way to be more collaborative and get more information out on what is important for other people. But why is it important when you run experiments?

[00:28:34] Because when you run experiments, you always want to be aligned with what the business wants and [what] the user wants. Let’s say we have the user needs covered with research. Sometimes there are different opinions in the business on what is important and a team can get pulled in many different directions.

[00:28:53] So in this type of situations, this book, Never Split the Difference, really helped me to probe 10 times better on finding out what’s the actual goals are. 

[00:29:04] This book tells us don’t ask why. Ask how are we able to do that? Because it can create defensiveness in people and engineers And this book really helped me to create much stronger relationship and move things forward way quicker with my team.

[00:29:23] Vipul Bansal: I quickly checked it on Amazon and added to my wishlist as well. Never Split the Difference. It sounds like really interesting book. I will purchase it once I finish these ones. So thank you so much. It was a fun conversation, Pius. Thank you so much for sharing your insights with our audience today. I’m sure the audience must have really loved it and, pulled insights for themselves and their organization. 

[00:29:49] Thank you so much once again, Pius, for speaking with us at ConvEx 2023.

[00:29:51] Pius Binder: Thank you too. Have a great rest of your day. 

[00:29:56] Vipul Bansal: Thank you.