Creativity in Experimentation: How to Get Better, Innovative, & Impactful Ideas

David Mannheim:

Yeah. Not bad, buddy. How are you?

Vipul:

Things are absolutely fine. I just request all of you to ask any questions that you might have during this presentation. Just send it to us for using the questions panel on the go-to webinar. And we’ll take them up.

And also, yes, since I’ve seen David’s presentation, I know that it’s very fun and filled with insights. So stay attentive and take notes. Over to you, David.

David:

Cool. Creativity. The experimentation is a topic that’s close to my heart, but as any narcissistic presentation should start, gonna start by talking about me. This is my timeline. Here I am, little cutie pie.

I was actually born in 1987. I’m not 35 as this timeline suggests. When I was younger, I always wanted to be a Disney animator. You know, the craft, the precision, the creativity, use of the draw, and what was known as animated cells, you had a foreground and a background. I visited Disney every year, I’ve been, an embarrassing 33 times to Disney World in Orlando. 

In 1995, Toy Story came out and ruined my life. I remember my dad saying, you don’t wanna work with those cells. Everything’s going digital nowadays. And I remember crying. And I didn’t want to work in digital. I wanted to be an animator.

I wanted to express my creativity. I don’t know what happened between 1995-2015. It was mostly beer or university, but despite not having once worked digitally in 1995, Fast Forward to 2015, and I own a digital agency. Fast forward another 5 years, and, you know, we create fabulous conversion strategies for the likes of Rudolph and Travis Perkins And Sports Direct, those conversion optimization strategies are ones that I’m proud of naturally, we do a lot of AB tests. And whilst I don’t get to do any hand-drawn animations, you know, that creative streak is kind of always stuck with me.

So that’s like a reason why I feel like I’m semi-qualified to talk about this, but I’m gonna go back to my love. I’m gonna let you guys know a little bit of a story. It only takes 5 minutes. I promise.

In 2000, Disney began to lose its creativity. While there were still a series of theme parks and occasional new rides, innovation was limited to imagineers who were isolated in the process of ride design. Copycat theme parks in Tokyo and Hong Kong emerged during that era, along with some massive flops. Notably, they created “The Lion King 1½” and “Hakuna Matata,” both of which were huge flops, as well as a slew of sequels that just didn’t work.

The parks themselves faced numerous problems. For instance, as you can see, Mr. Mike Wazowski over here is losing his arm. There were also issues with maintenance, incidents of children getting lost on a day-to-day basis, but the most significant problem was data. With personal computers on the rise (the iPhone debuted around 2007), data was everywhere. However, Disney lacked access to this wealth of information.

They had nothing but anecdotal data about what their users were doing in their parks. When you entered the park, it was like entering a black box. What were the guests actually doing? To address this issue, Mick Crofton, the then-president of Walt Disney World Resort, instructed a task force of five people—not Mickey, Minnie, Goofy, Pluto, and Donald, but actually imagineers.

And the task was to identify pain points. What were the barriers to entering the experience faster? They returned with observations such as theme parks with turnstiles or rides with long queues. However, they discovered that implementing one solution in one area could have a ripple effect across the entire park. How could a team of five address all the problems across a series of five theme parks? Well, it turned out that inspiration could be found everywhere.

While traveling between Disneyland in California and Disney World in Orlando, the team found inspiration in an unexpected place: Sky Mall magazines. These magazines, often found in airplane seat pockets, were infamous for their array of gadgets that passengers rarely purchased. However, the Fab Five drew particular inspiration from a wearable device featured in SkyMall known as the TriNZ magnetic band, originally designed to improve golf swings. They also encountered the Nike Plus Sport Band. This sparked an idea: what if Disney created something similar?

Thus, the MagicBand was born. This innovative band became the key to unlocking various experiences at Disney World. It allowed guests to access their hotel rooms, make purchases without needing to carry a bulky wallet, and even store FastPasses to skip queues. By reducing wait times, overall park satisfaction increased, leading to more time spent enjoying attractions rather than standing in line.

They’re doing more, they’re spending more. Can you imagine a scenario where you tap onto a ride and it breaks down? Disney notes that you had a negative experience, triangulates your position, and sends someone in to make up for your day, if possible. So, behavioral data is now available.

What an incredible tale of creativity! Removing ourselves from the day-to-day, our marginalization of theme parks, jumping on a plane, taking inspiration from another product, and applying it to your own. Now, that was all the way back in 2013. Unfortunately, I feel when it comes to A/B testing, we could learn a lot from that to apply it to what we’re discussing today. We’re not just talking about the problem of creativity with A/B testing, but also what to do about it.

Guys, it needs improvement. Come on, we’re all creative, raw human beings. We can genuinely do this. So, I have a hypothesis, naturally, being in experimentation. Unfortunately, I can’t statistically prove it. So, I don’t know if this is an assumption to believe. We start with best practices, and part of my French, share case studies. These are the standard, widely published, with well-known, sticky filters here, colored call-to-actions there.

Good listening. Those case studies and best practices are often limited, either to solutions that don’t necessarily tackle the problem, or they’re purely focused on usability. Rarely do they address changing the user journey or behavior. Because these case studies come with attached results, they’re often seen as quick wins, leading everyone to copy them. Over time, these solutions become the norm and are expected.

They neither impact the journey nor will they. So, that’s my hypothesis. Let’s go through that one by one. Who remembers Test 1? It was a series of case studies where you selected what you thought was the winner.

I didn’t show you the results. Is the blue button the winner, or is the orange button the winner? Those cases were really basic, dare I say, like something you’d see on an Optimizely or VWO webinar.

They’re the ones that started testing from the Obama campaign, experimenting with button colors, CTA copy, images, headlines — what we would consider very basic now. I think since Trump came in, they’ve stopped using that case study. These types of case studies are what we consider as best practices or standards. And here’s the thing: those case studies, those best practices, they’re limited to usability changes.

What do I mean by that? At User Conversion, we categorize our experiments into three classifications: usability, anxiety, and motivation. Usability involves making an action easier to achieve, like removing navigation links within a checkout to create a closed checkout. Anxiety is what prevents a user from taking an action, such as adding trust logos to reduce payment concerns. Motivation is what persuades users to take action.

So psychological techniques like scarcity or urgency are utilized to push users through a checkout process. On average, a motivational experiment is 103 times more impactful than a usability change, and anxiety experiments are 34 times more impactful. We know this because we’ve classified 789 experiments and found these statistics. In short, usability changes do not alter user behavior; they simply make tasks easier. Yet, as optimizers, we often focus on testing usability-based changes. 

Thank you, Obama campaign, for popularizing these ideas, which are now ruthlessly copied. If I were to tell you that we have a problem where users reach halfway down our product page and then struggle to find the checkout button, what would you do? I guarantee the vast majority would suggest implementing a sticky “add to cart” button. We’re conditioned, almost subconsciously, to gravitate towards these types of solutions because we’ve seen them work before or read about their success stories.

You’re likely familiar with Innocent Drinks, although they’re now owned by Coca-Cola and have expanded, they were known for their novel approach to copywriting. They’re quite cheeky and colloquial in their communication style.

And to such an extent that this tone, like, has become standardized, and consumers often expect it. The creator of this term is a chap called Paul Burke. If you Google him, he talks about his story. Interestingly, he actually regrets how this has taken off and how Burger King is copying this tone unnecessarily. Due to this copying, our users are becoming more and more desensitized, meaning our solutions have less impact. This is our attempt to add a method behind the madness.

I believe that our users are becoming more desensitized. I believe that our solutions lack impact because they are copied. Here are two usability experiments we’ve conducted over a series of four years, from 2015 to 2019, across different websites. Naturally, users are going to react differently to different stimuli. But whether it’s a closed checkout or a USB barter with certain unique benefits, we can observe a pattern of behavior. The impact of each experiment decreases over time.

In 2015, when we ran a closed checkout test, we would see north of a 10% uplift easily. Now, we’re lucky to see 3% or 4%, if anything. While it’s not statistically significant because it’s the same tests run over different sites, we can still discern a pattern. Where case studies are rooted and promoted in generic, basic executions, they are often copied from one site to the next. Consequently, users are becoming more and more desensitized to them.

Now, let’s dive into the juicy stuff. What can we do about this hypothesis, whether it’s true or not? I’m going to start with some examples and suggest we get creative. I have five tips on how we can be more creative with our experiments later, but first, I want to highlight some experiments that we’ve conducted for some of our clients.

The client, Biscuiteers, specializes in selling gorgeous biscuits as gifts for various occasions. One problem they encountered is that users often struggle to find the perfect product as a gift and become overwhelmed. They would browse the listing page, thinking, “Oh, I like that,” or “That’s a nice gift,” and so on. A typical approach to solving this issue would involve product recommendations or effective merchandising strategies. However, we took a different route.

We actually interviewed their staff and asked them, “Hey, David, you work in manufacturing. What’s your favorite biscuit and why?” By creating an emotional connection to a product through linking team members’ favorite biscuits to the user, we promoted this aspect. For example, we would highlight, “Congratulations! You’ve stumbled across David’s favorite biscuit.” We borrowed this idea from the practice often seen in cinemas, where before a film, it might say, “Hi, David, my favorite film is Avatar.” We observed a significant 16% increase in “add to cart” actions for similar products. This was a valuable learning experience for the Biscuiteers team.

They created little stickers featuring those favorites and replicated them in storage, which was quite sweet. About Travis Perkins, what do you do when users struggle to find products among the over-categorization of 24,000 products in over 600 categories? Less than 40% of users reach a product when using the navigation, and it’s even worse on mobile.

You could prioritize search or try to push users to use it. You could also consider redesigning the entire navigation, but that’s a daunting task. So, we actually took inspiration from the August catalog, those huge heavy books that could give you a hernia when flipping through to find the index or the A to Z section. We implemented something similar. When there are so many products available across numerous categories, why not create a comprehensive index of all those products? Following the phone book example, we implemented this and saw a significant increase in baseline conversion, a staggering 26%. Interestingly, 40% of all users preferred to use the A to Z navigation over the menu.

It’s amusing how we often get stuck in the structure of arbitrarily trying to find products and linking them back to categories. For example, in Travis Perkins, if you’re looking for decking, you might have to navigate through categories like outdoor, garden, timber, tint decking, timber wood, and so on. It’s like working backward in your brain. Hence, the A to Z approach worked really well for them.

Here’s another example: go to any e-commerce website, add a product to your basket, and what do you see? A notification congratulating you for adding Sebastian. Well done.

Now, what do you do? You click back. Why? Because you can almost guarantee that the one product you’re not going to add to your basket is the one the website leaves you on after adding it. Right?

Showcasing users recommended products related to the one they’re viewing, as well as similar categories, is crucial. For example, with Flannels, we observed a 43% increase in users viewing another PRP or a PDP. What’s interesting is that while we were getting more users to products, they weren’t necessarily the right ones. So, one of the key learnings from this was about personalizing products, especially within this scenario.

I could continue, but I urge everyone to challenge the status quo when it comes to navigation or menu structure, especially on mobile. It’s often considered merely as a structure, okay, nothing more, and few people think of it as more. But when we’re asking our users to cognitively associate a product with a category, like in the decking example I mentioned for Travis Perkins, it’s incredibly challenging. It’s like a guessing game, what’s behind door number 1? I call it the Babushka doll of guessing games because you’re constantly guessing what’s next.What’s behind door number 2? And so on. Let’s take it back. What’s the purpose of a navigation menu? It’s to aid product discovery.

What about adding recently viewed products, for example, in the navigation menu? Or adding recently viewed categories, depending on your business and products? Challenge the navigation. One of my favorite articles is by the Baymard Institute, though it’s quite geeky. This was ages ago, and they conducted a usability study on breadcrumbs. They challenged breadcrumbs, suggesting you need two different types: structural and contextual.

For instance, let’s say I’m on Sports Direct, and I search for tents. I land on a search results page and then click on a product, a cool tent. The product’s breadcrumbs might show something like “Outdoor Camping,” indicating the structural perspective of where the tent is, but not where the user came from, which was the search results. Usually, users just click back a lot, indicating a subpar journey. So, why not create breadcrumbs that offer to take users back to the search results? We implemented this for a client and saw an interaction rate of around 9%, demonstrating that users wanted that feature.

Now, let’s delve into a few more examples. But first, I want to discuss how we can all implement these ideas. Some are relatively simple, but they’re more about the mindset. I don’t want to get too theoretical, but let’s talk about the “5 C’s” I’ve developed. Yes, it’s actually 6 if you include culture, and yes, I did ensure all the words started with a “C.” The first three are cultural: conditions, conflicts, and collaboration, while the next two are practical: copy and continually iterate. It’s a bit of a stretch, but it still starts with a “C.” Let’s dive into them anyway.

These principles represent the foundation of creativity, at least within experimentation or solution design. Needless to say, if you work in an environment that’s not creative, output can be quite limited. Exercise, meditate, ensure your physical space is stimulating, and so on. These are all beneficial, but conditions for creativity go beyond just the physical environment.

One of the hypotheses I have—I was actually chatting with a friend about this guy named Max Hopkinson, who works for a great agency called Bind. He’s actually writing a white paper on creativity. Please reach out if you’re interested in reading about that. But we have a belief that the current conditions and demands of trade, coupled with work in sprints and trade pressures, lead to reduced risk. This results in a default to creating solutions that we know or that we’re conditioned to. You know, trade demands results yesterday; CEOs are under pressure; e-commerce product managers move away from risk toward comfort factors. The business environment we often see is a place where risk is avoided, and immediate action is demanded. You know, walk fast, make it happen, ship it, do it yesterday—this mentality prevails.

So, conditions are really important. Conflict—I highly recommend reading a book called “The Five Dysfunctions of a Team” by Patrick Lencioni. He talks about the pyramid of factors that constitute a successful team: trust, conflict, commitment, accountability, and attention to results. I won’t go into all of this, but trust is required as a foundation. However, the fear of conflict—fear to speak up, to contribute, to bounce ideas off one another—creates a fear of, I would say, creativity. The most creative solutions don’t come from a single source; they come from multiple sources.

This is the one that, personally, I struggle with—conflict. Because conflict is often associated with confidence, specifically, confidence in knowledge and in oneself. So, this is a difficult one to get right, for me anyways. Collaboration—obviously, it’s the most important.

You know, we set up User Conversion as a multidisciplinary team because we actually thought that was the most creative way to execute a campaign or a strategy—to get different people in the room together. And it’s a theory that’s shared by most, right? Or by some, by all, by most—this theory of T-shaped marketers joining together to create a Tetris roadmap. There’s a story about Elon Musk, and the reason why he’s such a smart chap is that he read hundreds of books when he was a kid on all different topics, from science to politics, or whatever.

But what’s really interesting is, in the 1990s, there was a psychologist named Kevin Dunbar who began conducting research on, well, researchers. He set up cameras in biology labs and recorded as much as he could, conducted interviews, and so on. Here’s the most striking discovery: it had to do with the physical location of where the most innovative breakthroughs for biological discoveries happened. It turns out, ground zero of innovation was at the microscope—or is it the conference table? Is it the water cooler?

Water cooler such an American term? The water thing, the cooler. But, you know, people would gather, basically, and discuss their work. So I think collaboration is really important. And, you know, we do that at UC. We work in sprints with our clients. And when it comes to collaboration, conflicts, and environment, radiating creative practices, we can really create some beautiful work together. So let’s go through this as an example. This is final again. And we asked you guys a question: what is the purpose of filters? Why do they exist? Why do users use them?

I’m about to say something really contentious. I’m so sorry, but the amount of e-commerce businesses that we speak to, and they all roughly say the same thing, which is: our filters are crap, we need to fix them. But when we actually demonstrate how many people actually use filters, it’s very, very minimal. Yet, often the solutions are trying to increase filter usage. So, sticky filters, me and my directors always take a bit because sticky filters seem to be such a poignant, very typical experiment, but we need to understand the behavior.

Like, why do they exist? So with vinyls, we ran a series of collaborative sprints where we presented evidence to the team. We found that it’s actually quite high—25% of all users use filters on the listing page, and that differs from mobile to desktop. Different attributes were used on different categories, etcetera. The most important finding was the “why”—why do users use filters and funnels? For Flannels, due to the abundance of products available, it was a choice. It was about narrowing down my options. And for reference, that’s not the only reason why users use filters; on different sites, their purpose changes. But we created a solution by reaffirming the number of products available, like 415 in that screenshot. We were able to almost add in to hit CT users. Hey, guys.

You’ve only viewed 21 products out of 450. You might wanna get it filtered a little. We tried, and it did work. Filter usage increased by about 15%, but more importantly, it increased accessibility usage, i.e., suggesting quality over quantity. But we didn’t stop there, and nor should you. This is about iteration as well. So we mixed that knowledge of the number of products with attributes.

We know which attributes work best and are most used in different categories. For example, coats and jackets—it’s brand, for sale, price, etcetera. Remember what I said before? Well, I’ll say it now: Generic ideas will yield generic results. It’s time to personalize user behavior for cohered user problems.

Given that we know a brand actually on coats and jackets highlighted filters in the product grid increased usage of filters by X. So we thought we would pop that brand filter, put it in a product grid. Hey, Cross Stone, 9% will use it, let’s run up, she’ll be like, well, you kind of expect that, you know, about prominence as well. We did see a 5% increase in conversion post-engagement. And then what do you do from there? Well, personalize those top brands that you see to the affinity of that user.

If I’m a user and I’m interested in Hugo Boss, Dolce and Gabbana, and Off-White, I might not be interested in Gucci or Stone Island. Or merge the two ideas together. You are 21 products into 550. Why not filter by brand?

Etcetera, etcetera. So I think collaboration is really important to talk to others because that’s where the best ideas come from. Conflict allows you to push those boundaries and even challenge the status quo, and the conditions in which you’re working—these are usually cultural—allow you to practice those types of things. There are two more Cs. Remember, there are five in total.

So one is copying. This is actually probably one of my favorites. So, what I don’t know if you guys know—I didn’t until I read it about 5 days ago—but what was the original inspiration behind GPS? It was Sputnik. It’s a really good story actually.

Accidentally, apparently, a few months after Sputnik launched, US scientists were able to determine the position of Sputnik from its frequency. The Department of Defense then said, hey, hang on, if we could flip this and turn an unknown location on the ground from a known location above it to try and glean where, say, nuclear submarines are? That’s GPS. So GPS was actually born from Sputnik.

It’s a really nice example of taking something—in this case, a satellite (I think Sputnik was a satellite)—and applying it to their industry. In this case, defense, military, and then adapting that even further for the purpose of recreational activities, like driving or watches or whatever it might be. And so, this is an example of the Travis Perkins 8-Zed Nav that I showed you earlier. Let’s go through a couple of these.

Let’s try and push those boundaries. Why don’t we copy Netflix’s browsing UI for ecommerce websites or product browsing? You know, it’s well known that Netflix has tested it. Users simply scroll in one direction. 

I completely appreciate that when you end up on Netflix, you might not end up watching anything. But I wonder if that browsing UI is there for a reason. Why don’t ecommerce sites replicate it? What about Tinder’s concept of swiping through items that we like or don’t like? Like that t-shirt? Swipe right. Don’t like that t-shirt? Swipe left. Similar concepts are continually browsing from one article to another. As you scroll down to increase dwell time, you’re reading an article on Medium, and it just naturally blends into the next article. Imagine that on a product page where you can scroll from one product to the next or a product listing page (PLP), sorry. I realize that I speak a lot about ecommerce websites. The majority of our clients are ecommerce. It’s quite natural for me.

Where was I? Using notifications as a form of social shopping. I wasn’t going to put this in after watching “The Social Dilemma,” but I did. You know, copying the likes of Facebook or Twitter by informing users, “Hey, a new product has been added while you were browsing,” or, “Dale from Cambridge has just purchased the exact same t-shirt that you’re looking at.” I mean, I get that it’s a bit Booking.com-esque, but these are all examples of copying from other industries or sites and applying their concepts to our site. They solve user problems for their users and their industries. We can do the same for similar user problems.

I always argue why we, as experimenters, don’t really look at, say, the pharmaceutical industry. They do quite a lot of experimentation, don’t they? You know, drug testing? Surely, they would be the experts. So, in terms of operations and practices, I wonder if we can learn a thing or two from them. Guys, I’ve got about 3, 4, 5 slides left.

So, if you’ve got any questions, now’s a good chance to write them out or take notes or whatever you prefer. But the last “c,” the dodgy “c” that I couldn’t really get a “c” for, is continually iterated. I saved this one for the end, but the only way to be creative is to let it evolve. So, creativity actually doesn’t often come in a eureka moment. You know, those moments are very rare. In fact, one could argue that they don’t really exist, that they’re either kind of preconditioned within the brain or they are a result of different stimuli over a period of time, which makes you think that’s how a moment exists.

Interestingly, there’s a story about Darwin. You know, he spoke of natural selection as this eureka moment, but there’s lots of evidence to suggest that Darwin had the full theory of natural selection written down in a notebook many, many, many months before he finalized his essay and published it. It was in “On the Origin of Species” or something. Anyways, so, yes, continually iterate. It’s really important. 

Let me talk you through this. I think this might be the last slide. But I implore you to iterate on experiments, to push that creativity and evolve. And at Flannel, it’s actually about 40 to 50% about experiments.

For iterations of one another. So this experiment addressed a common user problem: Flannel’s customers who were not signed into the site often couldn’t remember whether they even had an account, let alone their password. This was due to the low annual frequency of purchases, a common theme for those trying to log in. Around 20% of them forgot their password, but more importantly, there was a percentage who didn’t even know they had an account.

When users encountered this password error, the conversion rate dropped significantly, even across sessions. Again, this drop was somewhat expected, but we assumed that users knew whether they had an account at this stage, right? So instead of users thinking they didn’t have an account and trying to go through a guest route, only to realize later that they did have an account, we decided to do the hard work for them and tell them upfront.

That’s what we did. We created a really clean version where we informed users whether they had an account or not after they entered their email address. For example, “David@userconversion.com, and yes, you have an account. Please enter your password,” or “No, you don’t have an account. We’ll just check you straight through to the guest.” I believe Magento sites use a similar approach.

The result was that 13% more users went back to the previous page, and 4% of users bounced less, helping flow through the journey going forward. It might have been the execution, rather than the concept, that mattered. We reviewed session recordings and data ops to gain insights. We created a scenario where, instead of assuming whether users knew they had an account or not, we gave them a series of options: “Is your email address associated with a Flannel account? Yes, no, not sure.”

What was really interesting is that the “not sure” option was utilized about 10% of the time. 10% of users actively stated they were not sure whether they had an account. This drove checkout progression by a 15% increase in users going back in the journey and by 60% more importantly. We iterated on that a few times.

So in summary, create the conditions to be creative. They can come from the boss, the leader, the manager, or they can be a physical environment. But for me, it was the conditions, the conflict, and the collaboration that were perhaps most important. We’ve got to ignore best practices. For the CMOs and the e-commerce managers that are on the line, best practice is limited. Let’s put it that way. Not that it doesn’t work; I advocate that there are no such things as silver bullets. But best practice is limited. Look outside your industry. Don’t accept the status quo. Really challenge what is the purpose of x, y, or z? I’m trying not to be generic in our approach.

So I think, you know, web address, email address is there if you’re interested, but guys, that was like a 35-minute session on creatives. It’s about what I’m genuinely passionate about. I’m currently writing about it. I’d write a book if I could. Are any publishers listening? I’d be interested in that. But, yeah, I don’t know if there are any questions. I don’t know if I can see the questions.

Vipul:

Great. Thank you so much, David, for an amazing presentation. I was actually taking notes, and staying true to the theme of the presentation – creativity. I really, really like the idea of it. You know, that he shared with Flannel, wherein you showed that, you know, scrolling… sorry, what was that option? Did you view or did you view 21 out of, you know, X number of products already? Right? And that was quite innovative because I haven’t seen that. Right? And I, as a user of an e-commerce website, would definitely like to see something like that because, you know, it ends up in a bit of anxiety, builds up that anxiety when you’re not able to get to what you actually went out to look for. But that definitely helps in my search and to refine my search and to close the deal even faster. So that’s really creative. I’m sure the audience would have found it creative as well. That’s an amazing idea. I must say, I’m definitely going to take this to my team and show it to them, to take inspiration from.

David:

Well, let’s all copy it, and then it will be best practice. And then users won’t, will ignore it from there on in, then it won’t have an impact in about 2 years.

Vipul:

Right. So, yeah, that’s the problem, of course. But, of course, the early adopters will definitely gain at least marginal value out of it until the time it becomes a new normal, right? And then, of course, you have to keep refining your approach and keep coming up with more creative ideas so that you can always stay ahead of your competition. That’s perfect. So I do see a lot of questions have actually come in. Although I had prepared a few questions from my side as well, I give priority to the questions that are coming in from the attendees.

Right? So the first question is from Josh. He’s asking, how are you handling these tests across devices? That is, the A to Z flyout is very different on mobile versus desktop.

David:

Yeah, well, look, Josh, there’s a couple of layers to think of, like, an experiment idea as layers. You have the user problem, you have the concepts, and then the execution. In that instance, the user problem was users are struggling to neatly associate a product to a category. The concept is to create an A to Z listing. The execution is exactly what you’re speaking about there, Josh. Right? It’s the, what does it look like on mobile versus desktop? It is what it is. It’s what it looks like.

The concept is more important than the execution, and as an experimenter, we’re trying to prove or disprove a hypothesis. The hypothesis should be about the concept, not about the execution. So in this instance, an A to Z, actually, we did that test so long ago. Goodness me. But I think we created something like an iPhone scrollable thing, you know, when you go to your contacts on an iPhone and you see all the alphabets down the right-hand side, and you just go to “D” for decking, for example.

Whereas on the desktop, it’s a lot more visible. It’s been a scrollable div, etcetera extension. So I just separate the concept from the execution there.

Vipul:

Right. So that actually made me curious about, you know, I think this is a question that has been asked several times as well. Now, do you really need to have separate hypotheses for each of the devices? So if a buyer is buying from a laptop versus a mobile versus his, you know, tablet. Or does that need to be, or, you know, different hypotheses for each?

David:

Well, see, I’m gonna say no. Again, we just need to think about what is a hypothesis. A hypothesis is an assumption that you want to prove or disprove. It’s not something that you want to prove or disprove by a range or number or, like, a binomial metric. It’s to say, I believe, by creating an A to Z category, it will reduce the anxiety in users trying to find their products, for example.

The execution could be a button that says A to Z. It could be a page that is an A to Z page. It could be, I don’t know, a 3D augmented reality VR Super duper thing. It doesn’t matter, but for me, hypothesis is about proving or disproving a concept. The continual iteration bit is about iterating on the execution. I don’t know if you guys can still see my screen, but in this example of the Flannels login page, you know, the concept actually remained the same throughout. The execution just changed. And we could have got it really right here, but the execution I assume was a bag of crap because it didn’t have a call to action associated with it. I know you’re all thinking that. Don’t worry. So, yeah, in my mind, it’s not about having separate hypotheses for every device, it’s about having a hypothesis that proves the concepts, and then the execution is what you iterate on.

Vipul:

Got it. Oh, it’s generally because people are in different environments and different modes when they are on these respective devices, right? Maybe somebody’s just taking a break and they are not near the laptop, and they just want to quickly swipe through the list of items that they want to purchase. So they’re just candidly looking. They’re just casually looking. And so, you know, maybe treating a few things, maybe some sort of personalization there on the mobile device versus laptop could help. But again, that’s something that you have to test, right? And you have to be creative, follow the 5 C’s that were just mentioned.

David:

No. I may do. Yeah.

Vipul:

Right. Cool. So just quickly taking the second question, it’s from Jenny Moore. The question is, what’s your opinion on starting over? And redesigning a website from the ground up rather than iterating.

David:

Oh, Jenny, how long do we have? There are, like, there are too many factors at play. Right? There’s stakeholders, macro factors, more environment. There’s the boss that turns around.

It says we’re redesigning it tomorrow. I don’t think there’s, like, the perfect way, in my opinion, because I think it’s all contextual. But from experience, we’ve been involved in both sides. Been involved in redesigns where we experiment to iterate and then continually evolve the the site, for example, one of the dangers, like the people won’t often tell you if the practicalities of that will be of those of you who don’t do server side and you do client side testing, or of those of you who do client side testing, and then you have a dev team that don’t implement your experiments often, you tend to get like this Frankenstein of experiment on top of experiment on top of experiment. So that’s like a really good contextual example of, well, if you go down the iteration route, IR devs, generally gonna take an experiment and implement it live, or is it server side, for example, so you don’t have to, you don’t have to go about doing that. In the instance of redesigning through research or knowledge, I think it’s just important to remember gonna get a slap on the rest for this, but that’s just like theoretical stuff just because a user says they will do something called research, suggest x, y, zed.

It certainly reduces your risk and increases your confidence that X Y is able to work, but an experiment is one of demonstrating practicality. Improving concepts through data. Yes. This works. I can prove that.

So I don’t really have an answer for you, Jenny, unfortunately. It is all contextual. Feel free if you want. If you wanna give us an email or something, you could tell me about your scenario. Maybe we could try to help, but, yeah, it’s contextual.

Vipul: 

Right, it’s very contextual. And I think there’s a major role to play from leadership because it’s a big exercise, right? Starting over, a complete redesign process requires a lot of resources, versus just tweaking or iterating upon a few things. So definitely, you know, just check if your leadership would be very interested in starting over versus iterating. So that will definitely help. So it all depends.

That’s the unwanted answer, but that’s it. Great. So the next question is from Gabriel. Gabriel was interested in asking the question himself. Let me quickly search Gabriel in the attendee list so that I can switch on his mic. Okay, found you. So, Gabriel, I’ve unmuted you. Right. So you can quickly go ahead and ask your question within a minute.

Gabriel:

So, I work with Cheryl here in Brazil, and while we are brainstorming, a lot of tech restrictions appear. Since we know the websites very well, usually, we find it bad for the creative process to have these restrictions at this stage. I would like to know your opinion. Do you leave this tech part aside when discussing solutions, or do you think it’s good to address it during brainstorming?

David:

No. Can you imagine if someone turned around to Disney when they were creating the Magic Band and said, “By the way, you can’t do anything on the wrist or technology, the technology doesn’t exist,” or someone turned around to Elon Musk and said, “By the way, you can’t create electric cars that don’t exist yet.” It’s like technology will state technology, you know, IT implementation, dev, it can be restrictive. You know, that’s, I think, where the collaborative conflict, culture, conditions come from.

You know, it’s about being open and really challenging one another. In those scenarios, prove it, Gabriel. No. I’m really advocating MVP, minimum viable products. If you want to create a feature that allows users to look at the couch in their living room, for example, create a button that says, “Look at your couch in your living room live,” and try to determine the intent of that button. The button doesn’t need to do anything, you know, it’s a painted door, it’s a false door test. It can just say, “We’re really sorry. This feature isn’t available yet.” So you, again, go back to the hypothesis question.

You’re trying to prove or disprove a concept. The concept in this case is: would users like to see couches in their living room? I just assume so. And that button would help you get around those tech issues. It’s just, you know, creativity is about solving problems in an effective manner, really.

And you, Gabriel, you actually have a problem there. That problem is, I assume, your development team can be restrictive. How do you solve that problem creatively, Gabriel? Do you get them more involved? Do you get them less involved? Do you get them motivated? Do you push them? Read the Steve Jobs book of how he created this risk mentality. I forgot what they called it. Where he really pushed his designers to create. You know, he called it, like, the Steve Jobs reality distortion or something. So, yeah, try and tackle that problem using stuff that you’ve learned within this webinar.

Vipul:

Perfect. I hope you got the answer, Gabriel. I have an interesting question from Spencer. It’s a bit long. So the question is, how would you suggest speaking with stakeholders that are concerned about best practice when test results don’t show a positive impact? Maybe promoting further iterations. In this case, best practice is perceived as perceived by individuals without sophisticated justification of this best practice.

David:

It’s difficult. Look, the one thing I would say is that we are in a specific industry, in a specific field. We have a lot of knowledge about this industry and this field. We shouldn’t expect stakeholders to hold the same amount of knowledge that we hold about this field. Okay?

Just in the way that if we speak to a finance director or a CFO, we wouldn’t, you know, they shouldn’t expect us to know everything about how they operate or how spreadsheets or P and Ls work. So empathy’s really important here. It’s not where I’m wrong at all, you know, that’s where arrogance sets in. It’s about empathy and authenticity and transparency and really understanding the stakeholder that you’re speaking to. So get to know them.

It’s quite a lovely answer, Spencer. I’m sorry, but get to know them and understand them and where they’re coming from. Often people will tell you some quite malicious things like, “oh, just do it and hide the results. We’ve done that before,” or “just do it and show them that it’s wrong and manipulate the results,” or whatever it might be.

My answer is probably a little bit more emotional than that. It probably doesn’t answer your question, but try to understand where they’re coming from, and it’s a slow education. It’s not an arrogant one, it’s a patient one.

Vipul:

Right. Indeed, it is. You know, there’s no right answer to this, right? And that’s all, I mean, it’s where the concept of hypothesis comes into the picture. Right? And they’re definitely trying to avoid all kinds of risks, right? And best practices actually help you do that. To avoid all kinds of risks because, hey, there’s a bunch of other businesses who have tested this out, who have tried this out, you already know what the outcome could be.

You have that idea. Right? So it becomes easy to follow best practices rather than to, you know, try out and come up with something really bold and something very, very new and innovative. So I think the value is definitely in being innovative. That’s where the competitive advantage lies well.

Don’t you think?

David:

Yes. You know, the graph that I showed earlier, the anxiety, usability, anxiety, motivation, what you’ll find is that a lot of best practices are written with usability issues. And sometimes, that graph alone really resonates with people. So just a reminder, a usability implementation is something that makes an action easier. It facilitates an action that’s already in existence.

And anxiety is creating something that will prevent an action and motivation something that will persuade an action. And often what you’ll find is that user problems, either why, that anxiety and motivation classifications are the only way to attack a why or address a why, but the only solutions for a why, a usability-based problem is something that will facilitate something that already exists. So when you describe that like that, I’ve been in, like, a picture or 2 before, and sometimes it just clicks, and it really resonates with them. But the best way for me is to just understand, like, where they’re coming from, what that means, and how to connect with them.

Vipul:

Right. Hope that gives you some idea, Spencer. Great. So I think we’re nearing the 60-minute mark, and I have just one more question. And, I saw one other person as well, you know, curious about this. So, if you can just, you know, share 2 or 3 books that you would recommend first for an experimentation professional. And then, you know, any other book that you are currently reading or like to read.

David:

Yeah, let’s do it live. So, there you go. “Step by Step” by Tom P. Oh, terrific works. Great book. Fairly recent. Came out this year. Really good. As you can see, the pages are fairly new. So I haven’t really read much of it, but the guy knows his stuff. Interestingly, the first chapter talks about culture and why experimentation is important to businesses.

I don’t have the next one that I’m gonna recommend, but it’s a Disney one. I’m sorry. Ed Catmull wrote a book called “Creativity Inc.” Catmull’s like the ops director of Pixar. And he talks about, well, creativity, but his road to Pixar and what, how they tell stories and to that notes.

The book that I’m reading at the moment is “Fix Our Storytelling.” I’m only two pages in. It’s been a rough couple of nights. But, yeah, it’s, for me, I really love stories. I love drawing and sketching. I love trying to just crave new ways to really address user problems. You know? I’m just really not on any revolution here. I just love to just push the boundaries a little bit.

Vipul:

I’ve quickly taken note of that book because I haven’t heard of the book before, but it sounds good. And, but Dean sorry. Who’s off again?

David:

Oh, Dean Musovic. I think he actually works for a pixel.

Vipul:

Right. Because, I think, months back I came across this infographic, about how Pixar, you know, on the same topic our storytelling, and there were around 14 or 18 points, I believe, maybe more, about, you know, how to portray the character, how to how to tell the story that really encapsulates, you know, captures everyone’s interest that’s that’s really good. And I would be definitely interested in reading this book. We’ll see if it if they deliver it, in India as well. But, yeah, that’s that’s great.

And, it’s time to close this webinar again. Thank you so much, David. For your effort and, you know, taking all the time to share the knowledge, amazing presentation, you know, with our audience today. That’s really amazing.

David:

See you later.