All You Need to Know About eCommerce Conversion Optimization | An Interview with Tomasz Mazur

The following is an interview with Tomasz Mazur, a Conversion Rate Optimization expert, currently working as a consultant with Peaks & Pies.

Tomasz has several years of experience working on eCommerce conversion optimization and UX. He has previously worked with Zalando, Europe’s leading online fashion website.

Download Free: Cart Abandonment Guide

Tomasz answers our questions about conversion rate optimization from the perspective of eCommerce professionals.

Regarding the CRO Team and Sponsors

1) What does your ideal eCommerce CRO team consist of?

The team composition varies with the company. However, there are a few key members that every CRO team consists of (or should consist of):

• A CRO manager who looks over the entire program—from strategy creation to website analysis and A/B testing
• A design professional
• A developer who can create functional test variations

The scale of the work of a CRO team mainly depends on website traffic. The objective is to make use of the entire website traffic always. Therefore, the greater the traffic, the larger a CRO team needs to be. Zalando, for instance, had a sizeable CRO team. I cannot disclose the exact number but there was a large number of CRO managers working on the entire website traffic. The web analytics team was also huge, working on the minutest of details on the website such as product sorting algorithms. There was also a user research team that was in charge of qualitative research; the team used methods like usability labs, prototyping, focus groups, user interviews, and so on. As Zalando was operating in 15 markets and had a dedicated mobile website, the large size of the CRO team was justified.

2) Who should be the owner of a CRO program in an eCommerce organization?

The ownership of a CRO program should remain with someone from the higher management who understands the business impact of CRO. This person is the sponsor of the program. With the understanding of the business impact of CRO, the sponsor doesn’t necessarily have to exhibit knowledge of all the nuances of CRO. The sponsor could be the VP of Marketing or the VP of Product.

Sponsorship is necessary to ensure proper delivery of resources to a CRO team.

Regarding Coordination with Other Teams

3) Does a CRO team require help from other teams in an eCommerce organization?

One team that works closely with a CRO team is Marketing (especially, the customer acquisition department). The KPIs of the marketing and CRO teams are often aligned. The input of the marketing team is quite valuable—they know the product and the behavior of the customers.

The customer support team also plays a crucial role in highlighting pain points across a website. This team acts as a channel for customer feedback and helps a great deal in developing hypotheses.

A CRO team also needs to have the product team on its side. The product ownership mostly lies with the product team and their buy-in is essential. I know a few organizations where the product team has a “Conversion Lead” who acts as a bridge with the CRO team.

Regarding the CRO Process

4) What is the CRO process that you follow?

The process typically consists of the following steps:

• Studying the quantitative and qualitative data of a website
• Identifying problem areas on the website
• Developing hypotheses that aim to address the problem areas
• Creating variations per the hypotheses
• A/B testing the variations
• Analyzing the test results and sharing it with the team

Here is a graphic illustrating the process we follow in Peaks & Pies:

5) How do you build a hypothesis?

I use website analytics tools such as Google Analytics to look for optimization opportunities across a website, for example catalog pages, product pages, and the checkout page.

When certain webpages are identified for optimization, these undergo rigorous analysis. I first check if a webpage has the essential eCommerce features (such as a recommendation engine on the home page) or not. If a certain feature is missing, we have got an opportunity for optimization. I use heatmaps and click-tracking tools to find elements or functionality that require optimization. Gathering user feedback, taking help of usability testing labs, and checking out competitors’ websites are other ways of creating strong hypotheses.

I first check if a webpage has the essential eCommerce features (such as a recommendation engine on the home page) or not.

6) How do you prioritize A/B testing hypotheses?

I think a simple prioritization model like PIE—Potential, Importance, and then Ease—can work well for beginners. I personally like the model that is based on potential, confidence, and ease. This model gives a chance to the CRO team to take into account its past experience. A more sophisticated model can also include “political impact” as a criterion.

Moreover, prioritization needs to be a collaborative task to be successful. To estimate the “potential” of an A/B test, you might require the help of your marketing team. Similarly, the developers and designers can help you estimate the “ease” part.

7) What is the next step after hypothesis creation?

The next step is the design and development of a variation per the hypothesis.

However, before the variation is tested against a control, it must go through a thorough “quality assurance.” I think quality assurance is crucial for highly effective testing, but it is not emphasized enough in the CRO industry. You must make sure that all variations of an A/B test are free of bugs and issues.

Consider this: You are trying to improve a page that is already very optimized. You aim to achieve a humble 5% improvement in the conversion rate. If the variation doesn’t work for 5% of your traffic (because you forgot to optimize for mobile users or Firefox users), your test will invariably fail.

I think quality assurance is crucial for highly effective testing, but it is not emphasized enough in the CRO industry.

8) Is sales or revenue always the primary goal of your tests? Or do you look at micro conversions?

I would say the metrics or KPIs you are tracking should always answer your hypothesis.

In some cases, it is quite simple to track the revenue metrics (for instance, while adding new payment options on the checkout page). But in most cases, you have to track a combination of micro and macro metrics.

Download Free: Cart Abandonment Guide

9) How do you analyze your A/B test results?

To derive valuable learning from a test, we need to conduct a thorough analysis.

I look at both micro and macro goals to get a better context of the results. I also dig deeper by analyzing the test results for different traffic segments. For instance, I compare test results for new visitors versus returning visitors. You need to deal with a concept called novelty effect. Your returning visitors, when encountering a test variation, will recognize the changes on the page and might hold strong feelings about it (similar to how people strongly respond to major changes happening on Facebook or Instagram). However, new visitors will be unbiased with your test variation and will interact without any prejudices. Another set of segments that is relevant to eCommerce is mobile and desktop visitors. The behavior of both kinds of users can vary significantly.

I dig deeper by analyzing the test results for different traffic segments.

An analysis is always followed by summarizing the test results and recommending a plan of action. You check if the hypothesis is valid and whether you need to implement the winning variation or run a follow-up test.

10) If you find that conversions increased for new visitors but decreased for returning visitors, what do you do?

You need to derive learning from the test and realize why the difference in user behavior exists. For example, this could be happening because of an offer for your new visitors such as “10% off for new visitors.” While the new visitors would be encouraged to shop on the website, the returning visitors might feel like they are not being offered the best deal and feel cheated. Your next step would be to set up a system to identify new visitors and display the “10% off on first order” deal exclusively for them. You can additionally have a loyalty program in place for your returning visitors.

The goal of CRO is to not just have winning A/B tests, but also gain more knowledge about your users so that you can provide them a superior experience. Think about the bigger picture—the entire eCommerce ecosystem. If you find that your close-up product pictures work better than zoomed-out ones, you can communicate this idea to your display marketing team and help them create better ads. If you find that the “free delivery” offer works better than a “10$ free voucher,” you can share the knowledge with other teams such as customer support and products.

11) How important is a long-term A/B testing calendar for high-traffic eCommerce websites?

I think a long-term A/B testing calendar is essential for any kind of website. You need to have a prudent approach; clearly define all the tests that you can conduct, along with the time that each test is going to take.

Think of time as a resource. If you are not testing all the time, you are wasting opportunities to optimize your website (the same time that your competitors might capitalize to move ahead of you). With a long-term calendar, you can easily identify time slots in which you can fit quick A/B tests and utilize all your resources effectively.

You can also learn how to build a CRO roadmap for your organization.

12) How important is a knowledge repository of past A/B tests?

A knowledge repository is crucial to make your CRO program effective. It helps you know what works for your users (and what doesn’t) and helps you create better tests in the future. It is also used to introduce newcomers to the testing culture of an organization.

With every test you perform, it is important to document the details. You can start with a simple Google doc, and later get a sophisticated and comprehensive spreadsheet. Share the document with the entire CRO team so that everyone is on the same page and can avoid repeating any mistake. When the number of stakeholders is large, you can even think of a weekly/daily newsletter.

I usually archive my test results for every quarter.

Regarding the Nuances of eCommerce Conversion Optimization

13) Which eCommerce webpages do you think are key to a CRO program?

All of them. It’s important to go through the complete customer journey and find optimization opportunities. The most common customer journey path is from the home page to the catalog page to the product page to the cart page, and finally to the checkout page. Of course, there are other customer journey paths as well.

It’s important to go through the complete customer journey and find optimization opportunities.

However, a good CRO manager should always be able to identify low-hanging fruits. I have seen that the home page is the most tested page of eCommerce websites—mostly because it attracts the largest amount of traffic. Next on the list are product pages. These pages have a lot of traffic from channels such as affiliate partners and display ads. Third in the list are the register or login pages of websites.

14) With a large amount of traffic, how long do you run a test?

It is necessary to run an A/B test until it delivers a statistically significant result. However, with eCommerce websites, it is also important to consider the business cycle. For instance, the business cycle for fashion eCommerce is about one to two weeks. So even when a test delivers results in a few days, you need to run it for an entire business cycle to derive useful learning.

15) How many A/B tests do large eCommerce enterprises roughly run on a monthly basis?

That completely depends on the scale at which an eCommerce website is operating.

Large eCommerce enterprises like Zalando easily generate traffic of millions of visitors in a single day. For example, Zalando had traffic from numerous markets and it could allow even 100 tests to run at an instance.

16) How does a festival season or a “sale period” affect a CRO process?

I have a good amount of experience working with fashion eCommerce. I’d say that the sale period definitely has an effect on their CRO process. One of the main goals in a sale period is to clear out the inventory. This is when the principle of “urgency” is deployed heavily in CRO campaigns.

This is when the principle of “urgency” is deployed heavily in CRO campaigns.

For other eCommerce websites, the heavy traffic period can be markedly different. For example, I am working with an eCommerce enterprise that sells premium alcohol. The on-season period for them is winter when people tend to stay at home more and buy alcohol (for personal consumption or as a gift). As a result, this enterprise is focused on optimization activities more during the winter season.

Your Turn

What do you think about this interview? Do you have any questions of your own that you would like to ask Tomasz? Write to us at marketing@vwo.com