Follow us and stay on top of everything CRO

Creativity in Experimentation: How to Get Better, Innovative, & Impactful Ideas

Duration - 40 minutes

Key Takeaways

  • Understand the difference between usability, anxiety, and motivation: Usability makes an action easier to achieve, anxiety prevents a user from taking an action, and motivation persuades users to take an action. Motivational experiments are 103 times more impactful than usability changes, and anxiety is 34 times more impactful.
  • Avoid relying solely on best practices: While best practices can provide a starting point, they are often limited to usability changes and do not necessarily change user behavior. They can also lead to desensitization as users become accustomed to them.
  • Be aware of the diminishing impact of repeated experiments: The same tests run over different sites can show a pattern of decreasing impact over time. For example, a closed checkout test that once resulted in a 10% uplift may now only result in a 3-4% uplift.
  • Avoid copying solutions: Solutions that are copied from one site to another can lead to desensitization among users, reducing their impact. It's important to innovate and create unique solutions tailored to your specific audience.
  • Be mindful of user desensitization: As users become more familiar with certain practices or tones, they can become desensitized, reducing the impact of these elements. It's important to continually innovate and refresh your approach to maintain user engagement.

Summary of the session

The webinar, hosted by Vipul from VWO, featured David Mannheim, Founder of User Conversion, discussing the importance of experimentation, creativity, and problem-solving in business. David emphasized the role of leadership in decision-making and the need for a collaborative conflict culture. He also highlighted the importance of empathy, authenticity, and transparency when communicating with stakeholders about best practices.

He encouraged attendees to understand stakeholders’ perspectives and educate them slowly rather than imposing their own views. The webinar concluded with a Q&A session, where David addressed questions about dealing with tech restrictions during brainstorming and managing stakeholder concerns about best practices.

Webinar Video

Webinar Deck

Top questions asked by the audience

  • How are you handling these tests across devices? That is, the A to Z flyout is very different on mobile versus desktop.

    - by Josh
    Yeah, well, look, Josh, there's a couple of layers to think of, like, an experiment idea as layers. You have the user problem, you have the concepts, and then the execution. In that instance, the user ... problem was users are struggling to neatly associate a product to a category. The concept is to create an A to Z listing. The execution is exactly what you're speaking about there, Josh. Right? It's the, what does it look like on mobile versus desktop? It is what it is. It's what it looks like. The concept is more important than the execution, and as an experimenter, we're trying to prove or disprove a hypothesis. The hypothesis should be about the concept, not about the execution. So in this instance, an A to Z, actually, we did that test so long ago. Goodness me. But I think we created something like an iPhone scrollable thing, you know, when you go to your contacts on an iPhone and you see all the alphabets down the right-hand side, and you just go to "D" for decking, for example. Whereas on the desktop, it's a lot more visible. It's been a scrollable div, etcetera extension. So I just separate the concept from the execution there.
  • Do you really need to have separate hypotheses for each of the devices? So if a buyer is buying from a laptop versus a mobile versus his, you know, tablet. Or does that need to be, or, you know, different hypotheses for each?

    - by Vipul
    Well, see, I'm gonna say no. Again, we just need to think about what is a hypothesis. A hypothesis is an assumption that you want to prove or disprove. It's not something that you want to prove or dis ...prove by a range or number or, like, a binomial metric. It's to say, I believe, by creating an A to Z category, it will reduce the anxiety in users trying to find their products, for example. The execution could be a button that says A to Z. It could be a page that is an A to Z page. It could be, I don't know, a 3D augmented reality VR Super duper thing. It doesn't matter, but for me, hypothesis is about proving or disproving a concept. The continual iteration bit is about iterating on the execution. I don't know if you guys can still see my screen, but in this example of the Flannels login page, you know, the concept actually remained the same throughout. The execution just changed. And we could have got it really right here, but the execution I assume was a bag of crap because it didn't have a call to action associated with it. I know you're all thinking that. Don't worry. So, yeah, in my mind, it's not about having separate hypotheses for every device, it's about having a hypothesis that proves the concepts, and then the execution is what you iterate on.
  • What's your opinion on starting over? And redesigning a website from the ground up rather than iterating.

    - by Jenny Moore
    Oh, Jenny, how long do we have? There are, like, there are too many factors at play. Right? There's stakeholders, macro factors, more environment. There's the boss that turns around. It says we're re ...designing it tomorrow. I don't think there's, like, the perfect way, in my opinion, because I think it's all contextual. But from experience, we've been involved in both sides. Been involved in redesigns where we experiment to iterate and then continually evolve the the site, for example, one of the dangers, like the people won't often tell you if the practicalities of that will be of those of you who don't do server side and you do client side testing, or of those of you who do client side testing, and then you have a dev team that don't implement your experiments often, you tend to get like this Frankenstein of experiment on top of experiment on top of experiment. So that's like a really good contextual example of, well, if you go down the iteration route, IR devs, generally gonna take an experiment and implement it live, or is it server side, for example, so you don't have to, you don't have to go about doing that. In the instance of redesigning through research or knowledge, I think it's just important to remember gonna get a slap on the rest for this, but that's just like theoretical stuff just because a user says they will do something called research, suggest x, y, zed. It certainly reduces your risk and increases your confidence that X Y is able to work, but an experiment is one of demonstrating practicality. Improving concepts through data. Yes. This works. I can prove that. So I don't really have an answer for you, Jenny, unfortunately. It is all contextual. Feel free if you want. If you wanna give us an email or something, you could tell me about your scenario. Maybe we could try to help, but, yeah, it's contextual.
  • I work with Cheryl here in Brazil, and while we are brainstorming, a lot of tech restrictions appear. Since we know the websites very well, usually, we find it bad for the creative process to have these restrictions at this stage. I would like to know your opinion. Do you leave this tech part aside when discussing solutions, or do you think it's good to address it during brainstorming?

    - by Gabriel
    No. Can you imagine if someone turned around to Disney when they were creating the Magic Band and said, "By the way, you can't do anything on the wrist or technology, the technology doesn't exist," or ... someone turned around to Elon Musk and said, "By the way, you can't create electric cars that don't exist yet." It's like technology will state technology, you know, IT implementation, dev, it can be restrictive. You know, that's, I think, where the collaborative conflict, culture, conditions come from. You know, it's about being open and really challenging one another. In those scenarios, prove it, Gabriel. No. I'm really advocating MVP, minimum viable products. If you want to create a feature that allows users to look at the couch in their living room, for example, create a button that says, "Look at your couch in your living room live," and try to determine the intent of that button. The button doesn't need to do anything, you know, it's a painted door, it's a false door test. It can just say, "We're really sorry. This feature isn't available yet." So you, again, go back to the hypothesis question. You're trying to prove or disprove a concept. The concept in this case is: would users like to see couches in their living room? I just assume so. And that button would help you get around those tech issues. It's just, you know, creativity is about solving problems in an effective manner, really. And you, Gabriel, you actually have a problem there. That problem is, I assume, your development team can be restrictive. How do you solve that problem creatively, Gabriel? Do you get them more involved? Do you get them less involved? Do you get them motivated? Do you push them? Read the Steve Jobs book of how he created this risk mentality. I forgot what they called it. Where he really pushed his designers to create. You know, he called it, like, the Steve Jobs reality distortion or something. So, yeah, try and tackle that problem using stuff that you've learned within this webinar.
  • How would you suggest speaking with stakeholders that are concerned about best practice when test results don't show a positive impact?

    - by Spencer
    It's difficult. Look, the one thing I would say is that we are in a specific industry, in a specific field. We have a lot of knowledge about this industry and this field. We shouldn't expect stakehold ...ers to hold the same amount of knowledge that we hold about this field. Okay? Just in the way that if we speak to a finance director or a CFO, we wouldn't, you know, they shouldn't expect us to know everything about how they operate or how spreadsheets or P and Ls work. So empathy's really important here. It's not where I'm wrong at all, you know, that's where arrogance sets in. It's about empathy and authenticity and transparency and really understanding the stakeholder that you're speaking to. So get to know them. It's quite a lovely answer, Spencer. I'm sorry, but get to know them and understand them and where they're coming from. Often people will tell you some quite malicious things like, "oh, just do it and hide the results. We've done that before," or "just do it and show them that it's wrong and manipulate the results," or whatever it might be. My answer is probably a little bit more emotional than that. It probably doesn't answer your question, but try to understand where they're coming from, and it's a slow education. It's not an arrogant one, it's a patient one.

Reading Recommendations

  • Creativity, Inc.

    by Ed Catmull

    "Creativity, Inc." by Ed Catmull provides a fascinating behind-the-scenes look into the innovative culture at Pixar Animation Studios. Catmull, one of the co-founders of Pixar, shares valuable insights on fostering creativity and managing a successful creative team. The book explores the importance of fostering an environment where creativity can thrive while also addressing the challenges of leadership in a creative industry.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Vipul from VWO: Hey, everyone. Thank you so much again for joining this VWO webinar. I hope you and your family are safe inside safe homes. My name is Vipul, and I’m the marketing manager at VWO. I am the moderator ...