Taciana Serafim on Prioritizing Impactful CRO Metrics for Meaningful Improvements  

This interview is part of our ‘CRO Perspectives: Conversations with Industry Experts’ VWO Blog series. This series aims to showcase leaders who are not only adapting to the evolving CRO landscape but also setting examples for others to follow. 

Leader: Taciana Serafim Noqueli de Oliveira

Role: Product Management Analyst at Serasa Experian

Location: Brazil 

Speaks about: Conversion optimization • eCommerce • User experience (UX) • Product management • Business Strategy

Why should you read this interview? 

Taciania is a seasoned professional boasting extensive experience in sales, marketing, and growth. She thrives on challenges, injecting fresh ideas into business strategies with her strategic acumen and collaborative approach. Her journey includes exposure to diverse business models, and fostering continuous learning. Notably, she founded CRO Brazil, a thriving community with 500+ professionals focused on experimentation. As a partner at Kamelus Academy, an ed-tech specializing in CRO, Taciana, along with her partners, contributes significantly to Brazil’s experimentation landscape. In this interview, we delve into Taciana’s commitment to professional development, community building, and innovation, shaping Brazil’s experimentation landscape.

Professional journey and passion in CRO 

Taciana: About 15 years ago, I began my career in the digital world, initially working in a commercial area involving a lot of interaction with digital marketing. This initial experience sparked my interest in exploring the scientific method applied to marketing more deeply. Motivated by this interest, I decided to make a career change.

To dive into this new path, I joined an agency in Brazil. This environment provided a valuable experience, allowing me to work with methodologies and techniques that were fundamental for my professional development. However, I felt that something was still missing.

Driven by this feeling, I founded a community, CRO Brazil, which currently gathers more than 500 professionals specialized in experimentation. My mission became to help these professionals to improve themselves. In addition to the community, I am a partner at Kamelus Academy, an edtech focused on CRO, which I founded with my partners Gabriele Lavreca and Pedro Andrade. We have knowledge about experimentation in Brazil.

At Kamelus Academy, our work goes beyond teaching. We strive to connect professionals and celebrate innovation in experimentation fields. Through the community I created and the school, I can contribute to preparing experimentation professionals in Brazil, equipping them with knowledge and skills to face market challenges.

Memorable A/B tests and their impacts

Taciana: Once I conducted a test following our in-depth behavioral studies to enhance the number of subscriptions for a particular service. I chose to explore various formats for presenting subscription options to users. Horizontal options, which I believed could simplify users’ quick comparison of plans, and the vertical slider format, which I thought could be effective in providing more detailed insights into each plan.

We ran the test for 4 weeks, and it became evident that the horizontally formatted plan stood out as the grand winner. When compared to the vertical slider format, the horizontal table layout resulted in approximately a 15% growth in subscriptions.

Definition of CRO success and metrics to measure 

Taciana: In measuring the success of our Conversion Rate Optimization efforts, I place significant emphasis on a set of metrics that not only quantify the direct outcomes of our tests but also reflect how the culture of experimentation is taking root within our organization.

These metrics go beyond traditional conversion rates to include:

1. Winner/loser/inconclusive test ratio: 

This metric provides insight into the effectiveness of our testing hypotheses and the overall direction of our experimentation efforts. It helps us understand the proportion of tests that result in positive, negative, or neutral impacts on our objectives.

2. Rate of tests yielding new learnings: 

One of the core values of CRO is the continuous learning and improvement it fosters. This metric tracks the percentage of tests that contribute new insights into user behavior, preferences, or website usability issues. It’s a testament to the value of even those tests that don’t directly improve conversion rates but enhance our understanding of the customer journey.

I place significant emphasis on a set of metrics that not only quantify the direct outcomes of our tests but also reflect how the culture of experimentation is taking root within our organization.

3. Rate of tests generating shared learnings across departments (recommendations): 

This metric is crucial for fostering a culture of experimentation across the entire organization. It measures the extent to which the insights gained from CRO activities are shared beyond the immediate team, influencing strategies and tactics in other departments. It underscores the collaborative nature of optimization and its role in driving company-wide improvements.

In addition to these CRO-specific metrics, we closely monitor business-related metrics tailored to our company’s goals, with a particular focus on the Return on Investment (ROI) of our initiatives. These business metrics vary depending on the specific objectives we’re aiming to achieve but are essential for ensuring that our CRO efforts contribute to the overall success and growth of the business.

By balancing these two sets of metrics—those focused on the proliferation of a testing culture and learnings, and those tied to direct business outcomes—we can more holistically evaluate the impact of our CRO program. This approach allows us to refine our strategies continually, foster a culture of data-driven decision-making, and ultimately, drive meaningful improvements across the organization.

Common A/B testing mistakes to avoid

Taciana: A/B testing is crucial for conversion rate optimization, but incorrect use can lead to misleading interpretations. A common mistake is not setting clear objectives, which makes analyzing the results difficult. Testing multiple variables at once without proper design can prevent identifying the variable responsible for the results. Choosing an inadequate sample size, either too small to be significant or too large, wasting unnecessary resources, is also a frequent error. Moreover, overlooking seasonal variations can skew A/B test results, as can focusing on vanity metrics, which sound good but don’t impact business objectives. Another misconception is not repeating tests, which can lead to implementing changes based on unreliable data.

To avoid these mistakes, it’s crucial to establish a clear hypothesis and specific objectives before beginning. Simple tests, which change one variable at a time, help identify what truly impacts user behavior. It’s important to calculate the sample size in advance to ensure the statistical validity of the test. Planning tests considering seasonality and specific events prevent distortions. Prioritizing metrics that directly impact business objectives and repeating tests when necessary are recommended practices to ensure decisions are based on reliable data and effectively boost business performance.

A common mistake is not setting clear objectives, which makes analyzing the results difficult.

Building an experimentation culture for Brazil businesses

Taciana: In a landscape as dynamic and diverse as Brazil’s, embedding an ethos of continuous improvement and strategic experimentation can significantly propel business objectives forward in a meaningful and consistent manner.

Embracing a culture of optimization allows us to learn from past mistakes, ensuring they are not repeated in the future. This approach not only saves invaluable time but also streamlines the development of innovative strategies. It’s about creating a cycle of learning and growth that continually refines our methods and outcomes, making every iteration more informed and effective than the last.

By committing to such a culture, organizations can navigate the complexities of the Brazilian market with agility and insight, turning challenges into opportunities for advancement. It’s a journey that requires persistence, flexibility, and a willingness to experiment, but the rewards — enhanced efficiency, increased inclusivity, and stronger, more sustainable growth — are well worth the effort.

The future of A/B testing and CRO

Taciana: I believe that with the rise of artificial intelligence and machine learning, A/B testing will become more autonomous, capable of identifying complex consumer behavior patterns in real-time and dynamically adjusting the test variables to maximize effectiveness. Personalization will reach new heights, allowing for unique experiences that adapt to the individual preferences of each user, almost in real-time.

The integration of technologies such as augmented and virtual reality into A/B testing will open doors to assess how consumers interact with products and services in immersive digital environments, offering deep insights into the user experience that was previously inaccessible.

Furthermore, as consumers become more aware of their data privacy, A/B testing will evolve to be more transparent and ethical, ensuring that data collection and analysis are conducted with respect for user privacy and consent.

In summary, the future of A/B testing is bright, marked by an era of hyper-personalization, enhanced autonomy through AI, digital immersion, and a renewed commitment to ethics and privacy. We will be navigating a world where experimenting is not just about optimizing conversion rates, but about creating experiences that deeply resonate with each individual, transforming the way we interact with the digital world.

The integration of technologies such as augmented and virtual reality into A/B testing will open doors to assess how consumers interact with products and services in immersive digital environments.

Advice for CRO professionals and businesses

Taciana: For businesses that are starting in the world of Conversion Rate Optimization (CRO) and A/B testing, it’s crucial to begin with a deep understanding of their target audience, as this will guide the creation of more accurate hypotheses for testing. Starting with small and simple experiments can make managing and analyzing tests easier, allowing for progressive learning. It’s essential to establish clear Key Performance Indicators (KPIs) to measure the success of tests and to choose appropriate tools that fit the needs and budget of the company. 

Fostering a culture of testing within the organization encourages viewing tests as a continuous process of learning and optimization, not just isolated experiments. It’s also important to learn from each test, regardless of the outcome, using the collected data to refine future strategies. Prioritizing user experience ensures that optimizations contribute to lasting improvements in conversion.

Lastly, patience and persistence are crucial, as CRO is an iterative process that requires time to show significant results. By adopting a strategic approach from the start, businesses can ensure that their efforts in CRO and A/B testing are more effective, contributing to long-term success.

For businesses that are starting in the world of Conversion Rate Optimization (CRO) and A/B testing, it’s crucial to begin with a deep understanding of their target audience, as this will guide the creation of more accurate hypotheses for testing.

The impact of VWO on testing programs led by Taciana

Taciana: Over the years, I’ve had the opportunity to explore various A/B testing tools, each with its unique set of features and capabilities. Before Google Optimize announced its end, I was already navigating the A/B testing waters in agencies that preferred to use VWO. This choice was not random; VWO has always stood out for its comprehensive testing capabilities, from simple A/B tests to more complex multivariate ones and beyond. However, my real dive into VWO began about two years ago, and I currently use it frequently in projects for the CRO Brasil community.

Unlike other platforms I have used, VWO offers an incredibly user-friendly interface that does not compromise depth and functionality. This combination of ease of use and robustness has allowed me to set up, execute, and analyze experiments with far greater efficiency than ever before.

Moreover, VWO’s feature set goes beyond testing. The platform offers insight and feedback tools that enable a deeper understanding of user behavior, which is invaluable for creating and effectively testing hypotheses. This holistic approach to optimization has not only simplified my workflow but also led to more informed and impactful decisions.

In essence, the transition to VWO has been a significant upgrade in my testing approach. It’s not just about the tool itself, but how it has enabled me to be more strategic, collaborative, and ultimately, more successful in my optimization efforts. 

VWO’s feature set goes beyond testing. The platform offers insight and feedback tools that enable a deeper understanding of user behavior, which is invaluable for creating and effectively testing hypotheses.