Follow us and stay on top of everything CRO

eCommerce Optimization – Secrets from the Trenches. Real-life Test Examples to Help You Set Your Winning Strategy in 2021

Duration - 45 minutes

Key Takeaways

  • Keep Tests Simple: For lightly trafficked pages, it's best to stick to simple split tests. Multivariate tests can be too complex and take too long to yield results.
  • Patience with Results: If it's taking too long to find a winner in a test, there's not much you can do but wait it out. The 1,000 to 2,000 sessions threshold is the sweet spot for getting results in a reasonable timeframe.
  • Inventory Test Threshold: If you could run one test to improve the add-to-cart rate, try the inventory test threshold. This involves using custom code to display a message like "only 7 in stock, order now" when inventory drops below a certain level. This has been proven to work time and time again.
  • Optimize Product Page: The product page does the selling, so it's important to integrate elements like reviews, shipping expectations, guarantees, and returns information in a tasteful and effective way.
  • Use VWO for Testing: VWO is recommended over Google Optimize for its stability and additional features. It's important to ensure you have enough traffic before starting an experiment, with a minimum of 1,000 to 2,000 sessions for a standard split test.

Summary of the session

The webinar, hosted by Shanaz, features Craig Smith, the founder and CEO of Trinity, a leading e-commerce optimization agency. Craig shares his expertise in e-commerce optimization, focusing on the use of VWO to enhance e-commerce effectiveness and boost sales. He emphasizes the importance of reducing analysis paralysis, conducting micro-conversion measurements, and focusing on page-level metrics during experimentation.

Craig also addresses audience questions on topics such as deciphering key variables in page versions, the number of tests to run simultaneously, and the impact of testing on traffic. The webinar concludes with a Q&A session, where Craig provides practical advice based on his 20 years of experience in the field.

Webinar Video

Webinar Deck

Top questions asked by the audience

  • When you have created a new version of a page and changed lots of variables and when the new version doesn't work, how do you figure out which variables are the key ones to change?

    - by Nick
    That's a great question. So let's say we ran an experiment on a mobile product page, and it did not perform to what we expected. Frankly, without primary research, you really can't tell exactly what i ...t was. That's gonna be, like, do you and we'll have internal discussions. Like, do we wanna go the drawing board way and create an entirely new concept, or do we wanna build out additional functionality on the concept that did not beat the variation? It's very difficult to tell for sure. We can look at analytics and see, okay, did bounce rate get affected? Were there you know, pages per visit that was enhanced from this variation, but to get that exact functionality changed or to find out its impact on the overall test, it's pretty tough to decipher without doing primary research.
  • How many tests do you recommend running at a time? And how many tests in a given time period? Is there some sort of equation for how often to test based on traffic?

    - by Brett Croquet
    Yeah. So you don't wanna test too many aspects of your transactional funnel at the same time because you can kinda muddy the water. I like to do 2 to 3 tests is kinda like the sweet spot. Now when you ...'re a really large organization where you have a website with an architecture that's extremely deep and it's all these different sessions. That's one thing. But an e-commerce store where you have that category sub cap product cart checkout. You know, I would say 2, maybe 3. I like to maybe do, like, a homepage personalization treatment, like new visitors, returning visitors, customers, geo-targeting, that in parallel with maybe a category page test where you're trying to get more people to the product page and then maybe a cart test training more people to check out. Try to separate them into guardrails. Right? Like, don't have them overlap. So sometimes if you're doing a cart page and a checkout page test at the same time, the water can get a little bit, you know, muddied with understanding what's really working best.
  • I'm curious about what happened with RPV and AOV in the checkout tests with the free shipping bar.

    - by Soren
    That's a great question. I don't have that data point right here. But I would be shocked if a test didn't win. I mean, that's why one would include to kind of give an example for when you expect somet ...hing that does not come to fruition. But I do not have, at this point, you know, the exact AOB metrics associated with that experiment.
  • How do you utilize the funnel feature to optimize your site in VWO?

    - by Eugene Co.
    The funnel feature in VWO. So we use VWO as primarily the delivery mechanism. A lot of our reporting that you saw in the well, you might not have seen the beginning of the presentation. It's geared ar ...ound Google Data Studio. So we're taking information from VWO and essentially using it to connect to Google Data Studio. That way we can, like, slice it and dice it and visualize it better. I'm not ultimately in VWO day in and day out all our engineers and my analysts are. So I can't speak to the specifics of that functionality. But we usually use from our visualization standpoint, Google Data Studio which plays really, really nice with VWO. If you haven't seen Google Data Studio, it's a free product. It just allows you to kind of, again, use multiple sources. So if I wanted to say in my VWO test, wanna look at the micro conversion rate, the macro conversion rate, the balance rate, and the pages per session. I could do all of that kinda get more of a global feel of what the impact was versus just one metric. That's one of the reasons we use it. But, I can't speak specifically about the final functionality, but maybe you can share.
  • Can you talk to us about statistical significance and what role that should play in selecting a winner?

    - by Greg
    Yeah. Sure. Of course. So statistical significance is when the math tells you that the variation is conclusive. You never wanna stop an experiment until you get that green light, or maybe red light. ... 95% is usually the percentage we look for. So when our analysts see 95% statistical significance in an experiment, that's when we stop/pause the test. Go to the customer. Let them know. And then at that point, we usually flip the switch to 100% of VWO while we hard code it into the active. So 95% is really the number you wanna look for.
  • Even with large sample sizes, how does VWO select a winner version? And what do you do when it takes too long to find a winner?

    - by Gustavo
    Yeah. So we don't use Google Optimize. We've obviously found VWO as a superior product you know, it's got much more bells and whistles. It's much more stable. You always wanna make sure before you sta ...rt an experiment that you have enough traffic to test. If you're not getting 2000 sessions on this test, you're probably gonna sit and take too long. It's gonna be more frustrating. So use, like, at least 1 to 2000 sessions for a standard split test, but, something to consider is people come to me and like, oh, we wanna do a multi-variate test on this page over the page. Let's get a lot of traffic. You know, multi-variate test with let's say four aspects of the site with 4 variations a piece is 4 times 4 times 4 times 4. It's gonna be, you know, a ton of variations. You will wanna do simple split tests on light traffic pages. That's the takeaway. An A/B test at max, an ABC test, but hopefully that answered the question about the traffic. And when it takes too long to find a winner, there's not much you can do. You know, you kinda have to wait it out. That's why that 1 to 2000 sessions threshold is really kind of the sweet spot.
  • If you could run just one test to improve the cart rate, what would that be?

    Great question. If I could do one test for add-to-cart rate, I would try that inventory test threshold. Because it's worked time and time again. I mean, literally, I had another 2 examples that my ana ...lyst gave me, but I didn't include it because it's the same experiment. Didn't wanna bore you guys. But, basically, what you're gonna be doing is if you're on Shopify, if you're on BigCommerce, if you're on Magento, there is a data piece that you can grab. It says how much inventory is in the platform for that SKU. And then use custom code that says, okay. When this drops below 10, when this drops below 7, whatever that threshold is, tell the user only 7 are in stock order now. You know, that has this work time and time again. It's not a very, you know, tough experiment to deploy, and that's a real quick and easy one. There are a lot of other ones as well. I mean, making reviews higher up is critical, hitting upon shipping expectations is critical, making sure that you talk about guarantees and returns is critical. The product page does the selling. So all those pieces need to be very tastefully integrated in that page so you're gonna speak to that user in that type of capacity.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Shanaz from VWO: Hello, everyone. Thank you for joining us today. My name is Shanaz, and, I’ll be your moderator today for this webinar. For those who are hearing about VWO for the first time, VWO helps you identify ...