The Testing Maturity Model Used at Dandy

[NOTE: This transcript is pending review and, hence, may contain errors. The final version of the transcript will be available soon.]

[00:00:00] Sara Pion: Hello, everyone. My name is Sara Pion. I am the head of marketing strategy at Dandy and today I’m going to be talking about conversion optimization at Dandy and how we think about it. 

[00:00:19] Just a quick heads up. This is a pre-recorded session. As we speak, I am currently on a plane across the United States but I will have my social channels linked in the final slide.

[00:00:28] So if you have any questions or comments or you just want to nerd out about conversion optimization, please don’t hesitate to reach out. 

[00:00:34] I love talking about this stuff. So let’s jump in. All right. So when I joined Dandy and still to this day, we rely really heavily on paid channels to drive traffic to our website.

[00:00:49] Facebook and Google ads are main driver as well as content syndication and paid syndication from our vendors and our partners but our paid team described our website as a content desert. 

[00:01:02] Even if someone landed on a specific paid offer page and they tried to navigate through the rest of our website, there wasn’t a lot for them to convert on as they moved through our website and that reflected poorly in their numbers and so I was brought on to help with conversion optimization.

[00:01:18] But at the same time, we started really heavily investing in content and webinars to make sure that we started to have offers that we could run on our website. 

[00:01:25] So if you’re in a similar spot where you’re just running ads to a demo form. We invested in content at the same time that we were investing in conversion rate optimization so that we were able to do more organically as well as getting more from the traffic that we were paying to come to our website.

[00:01:44] So our overall program goals at Dandy for conversion optimization is number one to increase website conversion rate blended. So that’s content downloads, webinar registrations and demo requests.

[00:01:56] We don’t just focus on one point of conversion. We want to influence every step of the funnel with our testing because that’s how we run paid as well.

[00:02:03] We have lead gen offers. We have MQL offers. We have SQL offers and then we have Opp creation and closed offers and we want our website to reflect that as well. We also want to inform messaging and positioning through testing. 

[00:02:18] PMMs know that messaging and positioning work is rarely done. Once you do it, you kind of have to start again because the landscape changes and your company changes and the world itself changes and you have to iterate what that looks like and what that means and how that resonates with your audience.

[00:02:34] We also want to personalize our website through our persona framework. There isn’t one kind of person who comes to our website and we want to make sure that our website reflects that and personalizes the experience based on who, what we know about you, who you are, what your motivations are, what your pain points are.

[00:02:51] And then from a UX, UI and SEO perspective, we want to inform site structure and design. How should people navigate through our website and what should it look like? And what should they see first? And what should they see second? 

[00:03:03] So all of these factors are our goals from a conversion optimization perspective and it spans across almost every single team at Dandy from a marketing perspective.

[00:03:16] So we help with increasing the overall conversion rate from demand gen programs but we also help product marketing and customer marketing understand how our customers in our prospects want to be spoken to and then from a design perspective, what should our website templates look like? And from an SEO perspective, what should our information architecture look like?

[00:03:33] And how should it be presented? Because people want to see one thing versus another and how we came up with this strategy was from a few different ways. Both quantitative and qualitative. 

[00:03:45] We looked at our Google analytics. We looked at our website performance. What are our highest traffic pages? What are our highest converting pages as well as our paid search data?

[00:03:54] What are people searching for when they come to our website? What do they want to see? What do they want to know when they land on our website? And are we presenting that to them? And then also our funnel analysis. 

[00:04:03] What are the most important pieces of content that move someone from an engaged lead to an automation qualified lead and then an automation qualified lead to a demo request lead.

[00:04:13] From a demo request to sales accepted etc. From a qualitative perspective, we want to know how do people engage with our site.

[00:04:21] How from a heatmap and eye tracking perspective, what are our competitors websites look like? And is that the industry standard? And should we question that? 

[00:04:29] Or should we align our website to be within those industry standards as well as running usertesting.com UX and UI test to see, does it make sense on how we’ve set up our website from a navigation perspective?

[00:04:40] Does it flow or are people not really catching on at all? From an audience insights perspective. We relied really heavily on the persona work done by our product marketing team. 

[00:04:51] So what are those persona insights? And what are those demographic insights? And what have they gathered from the interviews that they’ve done both on the product marketing side, the product management side, design side and test that we can run ourselves as well? 

[00:05:03] What do we know about our audience that we can make sure is presented to them when they come onto the website? And then our testing strategy is defined in a road map. 

[00:05:11] So we like to forecast out our test for two months and then the test that we run within those two months informs the next two months.

[00:05:21] So the fun thing about conversion optimization is that your job is never done. Everything that you do can inform what you do next because once you run an A/B test, you change the base site, you start all over again and then we also have defined audiences and defined pages that we run tests on and we run tests for.

[00:05:40] So this is how we’ve come up with our entire strategy and how we will run tests now and moving forward. If you do not have a CRO strategy, that is okay. Here is where we started and here’s how you can start as well. 

[00:05:58] What are the goals you want to influence? And that can be on site behavior to start so that you can prove the concept of commercial optimization to your executive team.

[00:06:09] So can you inform on site behavior to start? Can you run tests that influence page views per session or bounce rate or time on site?

[00:06:19] And from there you can show the actual value of like, hey, we ran this test and we saw that our bounce rate decreased by X percent and people actually spent more time on the page and ended up converting at a higher rate.

[00:06:29] From there, can you start to inform overall website conversion rate? Can you run tests that just increase the overall blended conversion rate from your website? 

[00:06:39] And then as you progress even further, can you try and drive specific conversions from specific personas and really get to that almost one to one or one to few level test on your site. 

[00:06:52] From a page perspective, what pages should you run traffic on? I always like to say that the pages that you run conversion optimization experiments on should have at least a thousand sessions per month. So you have a critical mass of traffic to get answers to your questions as fast as you possibly can.

[00:07:10] If you don’t have any page on your website that gets a thousand sessions per month, just choose your highest traffic pages or your highest converting pages and start your testing there. 

[00:07:21] But don’t forget the 80-20 rule. 80% of your conversions are going to come from 20% of your pages.

[00:07:26] So you shouldn’t run tests on every single page on your website. That’s not a good use of your time. It takes up a lot of your time. It’s very tedious. Don’t do it. 

[00:07:34] Instead, pinpoint those pages that have really high traffic but low conversions are really high conversions, so you can try and influence the testing and the overall conversion rate of those pages with your testing.

[00:07:49] Another 80-20 rule is how much of your traffic should be in a variation audience versus how much of your traffic should get your base site should be the control and to get the most out of your CRO testing and to get answers as quickly as you can. I like to say 80% of your traffic gets variations.

[00:08:07] 20% of your traffic should get the control. This is to make sure that you’re also running like a clean test and you can say with confidence that the test that you’ve been running have influenced the conversions that you’re seeing. 

[00:08:20] From an SEO implication, when you’re running conversion optimization experiments, you’re generally using tools that act as an overlay on your website, which means that you’re not changing the way that Google indexes certain pages or the way that Google is understanding the site architecture of your website.

[00:08:39] Once you start making changes to your base site, that’s when the SEO implications start to kick in but when you’re running tests, you shouldn’t see any SEO implication besides better onsite behavior. Collaboration. 

[00:08:51] So as I mentioned in the slide before or two slides before CRO is not a single player sport at Dandy and it shouldn’t be a single player sport in your organization either.

[00:09:01] It is incredibly collaborative. It impacts almost every single team, not only in your marketing team but across your business and so make sure that you are collaborative. 

[00:09:11] Communicating properly with the right people but also getting insights from people who might have answers that you need. So first things first marketing designers, making sure that the visual tests that you run are approved and pretty and look good and are using, approved designs.

[00:09:28] Marketing designers generally also have some sort of like UX and UI background. So if you have less experience in that realm, relying really heavily on these designers can be helpful for your first few page layout experiences. 

[00:09:41] Product designers are also a really great partner when it comes to telling your product story and how you want to tell your product story across the website because they’re super familiar with how customers use your product today.

[00:09:52] You can kind of start the onboarding process early on your website through getting an understanding of from your product designers, what kind of story you want to tell, what features you want to highlight, what are the most important parts of your product that your users like get really emotionally attached to make sure to highlight those on your website. 

[00:10:13] Product marketing, of course, you want to make sure that you’re partnering with them. So any copy changes you make are approved from a positioning standpoint but also they’re just a great partner in allowing them to test messaging on the site so that they can get at scale.

[00:10:27] An answer to which positioning statement works the best for our entire audience and then product management is also just another great collaborator when it comes to telling the product story and telling the product story in the context of your customers, what is their pain, what are they feeling, what tools are they looking for to solve their problems and then leveraging that to tell the story on your website.

[00:10:55] How we structure tests? So, we don’t just run tests and don’t tell anyone. We usually like to come up with a campaign brief because that is how our marketing team communicates their campaigns. I always like to add these three things to my campaign brief. 

[00:11:10] So it makes it super, super clear that we are running a test. This is our hypothesis. This is the background information that powers that hypothesis and this is what we think will come from this test. 

[00:11:19] Then I like to use the campaign brief as a house for like our mock ups, copy change variations, any inspiration that I’ve seen on other websites that we can use for mocks that may not yet be created and then also updating that campaign brief with results once the test ends. 

[00:11:37] So it’s a living document. So I always start with a hypothesis. When a certain persona comes to the site, they should understand blank so that they can blank, which will lead to blank. 

[00:11:48] That is generally how I like to structure my hypothesis so that it makes it super clear that this is the audience that we’re targeting and this is what we want them to understand so that they perform this specific action.

[00:12:00] The background information should be research that powers your experiment. So that means that you shouldn’t just be running an experiment because you want to really do some research into the experiments that you’re running to show that there’s going to be a business impact on these experiences and this experiment itself.

[00:12:16] There can be multiple different experiments in a project plan or a campaign brief. For example, we want to increase the overall conversion rate of our homepage. So we’re going to run hero image variations, page layout variations, headline variations and on page copy variations and some of those are going to be run as multivariant.

[00:12:34] Some of those are going to be running strictly A/B and we’ll get into those kinds of tests real quick but just as a little FYI, your campaign brief doesn’t have to be about each individual experiment you’re going to run but a full project plan of like, this is the conversion rate we want to influence and this is how we’re going to do it and it’s going to be done through a few different ways. 

[00:12:58] So moving to the forecasted lift, this can kind of start out as an educated guess. You can say we’ve never run a test like this before, so we’re going to shoot for a certain conversion rate and see, how close we get. 

[00:13:10] But as you mature in your testing and as you start to perform similar tests, you can use that. Those results from those previous tests to influence the forecast for tests similarly structured as you would when you’re forecasting any number in a demand gen funnel, we grew 5% month over month.

[00:13:31] The last two months, we can assume that next month will grow 5% in this specific area based on the campaigns we’re going to run etc and finally, how to measure success. 

[00:13:42] So conversion optimization, CRO experiments should be measured in success through statistical significance. 

[00:13:48] When you’re running really crucial tests, we want to have 95 to 99% statistical significance and how you calculate this, as you plug in the sessions of your variations and the number of conversions that you’re seeing, the sessions for the control and the conversions that the control saw and your A/B test calculator.

[00:14:07] Your tool will calculate if you have reached statistical significance and at what level you want to get 95 to 99% statistical significance because that allows you to say we are 95 to 99% confident that the lift and conversions we are seeing is due to our experiment and not another external factor. 

[00:14:26] That helps with the naysayers or the executives who are like, are you sure? It is a way for you to kind of ensure that what you are doing is really driving impact from a statistical perspective.

[00:14:39] We actually drop our confidence down for non crucial tests. So for things like testing messaging and positioning through on page copy variations or even headline variations. 

[00:14:48] And just like small image changes. We will use 80% confidence because it’s not anything that will change the overall workflow of our team or pitch that our sales team uses or anything like that.

[00:15:01] It just is more of like a validation like, okay, we thought this would work and it did great. Let’s make some big site changes. 

[00:15:11] All right and now let’s get into the four levels of CRO experiments that we think about at Dandy in kind of a maturity model. 

[00:15:19] So starting first with testing and optimization, these are A/B tests and multivariate tests and you can spend your whole CRO career just running A/B and multivariate tests because there is so much that can be done here. 

[00:15:34] So, quote unquote, level one perspective, this can be incredibly mature. This can be really impactful to your business too. So, level one is not in any way, shape or form something to be ashamed of.

[00:15:49] There is so much that you can do. Just A/B testing and multivariate testing and so many questions you can answer through these kinds of tests. 

[00:15:57] So, A/B tests are changing and testing a single element versus another. So three different headlines on a specific page or four different hero image variations and you’re testing the hero image variations against each other. 

[00:16:11] Different CTA text variations. Get a demo. Get started. Fill out an interest form etc. All of those would be considered an A/B test or changing the on page copy. Multivariant test is testing a combination of multiple element changes.

[00:16:26] So headline variations and hero image variations. So you’re seeing the second headline variation with the third image variation and does that combination increase conversion rates of the page? 

[00:16:36] CTA text variations and on page copy variations. Do those two tests together increase the overall conversion rate of a page.

[00:16:46] So some examples that we’ve done at Dandy, we use ConvertFlow to run, nav variations and embedded CTAs and pop ups and sliders. So we have sticky nav offers and embedded CTAs and sliders that all A/B test different offers. So one offer versus another and that is it.

[00:17:07] It’s not a combination of StickyNav plus Embedded plus Slider. It’s just each element in its own individual experience. Also the visuals that we use dark moded type visuals versus light mode. Different background images, all of that is A/B tested. 

[00:17:27] We change the CTA colors of our website based off of our brand guidelines. So we use a specific color as our primary CTA text color or a CTA color. 

[00:17:35] But what if we use one of our secondary colors and does that change the overall conversion rate and should we change the colors across our website of our CTAs because we understand that people prefer gray buttons over blue etc and then go digital is an offer page we have on our website. 

[00:17:53] So if we change the headline to highlight the free scanner that we give you or to highlight the fact that we are a national digital dental lab, what does that change? When people engage with our website the multivariant tests that we run were on our homepage.

[00:18:10] We tested different headline variations with different CTA text variations with different hero image variations. These were all tested against each other to come up with the best combination of the three as determined by our traffic. 

[00:18:21] So it’s like a little recipe. If we add this plus this plus this, what’s the best combination of those three? Moving on to level two, which is all about targeting. 

[00:18:31] So you’re targeting people coming to our site based on specific conditions. So, certain sources or certain behaviors. So are they coming from email? Are they coming from paid search or paid social? What are the UTM parameters in those URLs from email, paid search or paid social?

[00:18:47] Are they coming from a certain campaign? And can you ensure that there’s like post click relevancy when people come to your website? Are they coming from organic and direct versus a paid channel? And what do you want those different experiences to look like? 

[00:19:02] Are they coming from social media? Do you have a huge social presence and you get a lot of people coming in from Twitter or LinkedIn or Facebook or Instagram?

[00:19:08] Or do you have a really strong backlinking strategy and you get a lot of referral traffic and how can you greet that referral traffic differently because they were on one page on the internet and now they’re on a different page and how can you make sure that context carries over from their site to your site or from a behavioral perspective.

[00:19:25] Is this their second visit or is it their first visit? Should you say hi welcome or should you say welcome back? Are they on mobile versus desktop if they’re on mobile? 

[00:19:34] What if you remove images from your pages and just have the text that’s most important or you remove text and you remove different elements to make sure that the mobile experience is actually a lot more seamless. 

[00:19:47] Do they convert on the last visit or have they been here a lot of times but they haven’t converted. Level 2 testing we’ve done at Dandy has been really specific on the sources.

[00:19:57] So we’ve done specific headlines on paid offer pages for different states. You’re coming from New York, you just search for best dental lab near me and we say we are the best dental lab in New York for example. 

[00:20:08] Specific headlines on webinar pages for paid traffic. We test out different ad copy on our webinar, paid ads. Can we make sure that ad copy is reflected when they come onto our website and similarly for syndication partners, different headlines and on page copy for syndication partners and this is all leveraged through the UTM parameters in the URLs. 

[00:20:31] So first party data they have, they told us who they are and where they’re coming from or Google did. Level three is segmentation. So this is when you start to incorporate first and third party data. 

[00:20:44] So, this can include things like Clearbit or Sixth Sense informing different segments on your website from a rules based perspective or using your own marketing automation platform and the cookies that are included in those to understand the different smart lists or segments coming to your website that you’ve already pre-identified.

[00:21:04] So, we use Marketo so we could greet customers based on their previous order history. We know you just ordered a crown, make sure you place your next crown order or we’re running a promotion. Your next crown order is buy one, get one whatever or their life cycle stage. We know that they’ve been a customer for three years but they haven’t made an offer or may place an order in three months.

[00:21:29] We want to ask them what’s going on with that or they’re in training and we want to make sure that they log in to complete their training or they finished training but they haven’t placed their first order yet. 

[00:21:39] What is the experience we want them to have when they come onto our website versus prospects? If they registered for our latest webinar but didn’t attend, can we service up that webinar recording for them when they come to the website? 

[00:21:49] Or do they speak to us at our latest trade show or are they a lead or are they an automation qualified lead or are they a marketing qualified lead? How can we personalize the experience for them?

[00:22:03] Do they fall within a certain paid ad segment? So are they part of our like ICP retargeting ad segment? We want to kind of roll out the red carpet for them when they come onto our website. 

[00:22:11] This is one to few kind of experience with segmentation and here also segments based off of previous web activity that kind of goes back to a webinar registrants versus attendees but also did you look at a pricing page but not convert, what is that kind of experience?

[00:22:27] It’s a high intent page visit. What do you want to do when they come back to the website that haven’t converted yet? You also have rules based segmentation. 

[00:22:35] So net new visitors who match certain criteria, pay traffic from New York referral traffic from a syndication vendor or you’re using the IP lookup kind of tool and you know that they are in a certain industry and in a certain business unit. 

[00:22:52] So, they’re in the healthcare segment and they’re an enterprise company. What do you want their website experience to look like? And finally, level four is one to one personalization.

[00:23:05] So this can be done as custom single page experiences. You have meetindy.com/Sarah or my company whatever and it’s unique and personalized from a one to one perspective. 

[00:23:18] It’s a specific landing page or a specific onsite pop up or if you use chat, you have specific chat messages targeted to that company or that person’s email and it’s really made for them or you have a custom full site experience. 

[00:23:33] Using first and third party data again to recognize the user targeting those VIP visitors and using the rest of your site visitors as a control. So that you can review the full funnel impact of creating this one to one full site experience.

[00:23:46] Because what happens if they come to your website and they don’t land on the unique landing page, you’ve created for them. 

[00:23:51] You’ve got to continue to roll out the red carpet for them and what is that full funnel impact when you do something like that? 

[00:24:00] So those are the four levels of conversion optimization and we kind of talked about statistical significance and seeing that as success for your experiments but what happens after that?

[00:24:09] How do you continue to inform your testing roadmap once you’ve reached the statistical significance, first things first, change the base site or don’t. Statistical significance can either be like, yes, we are a hundred percent confident or 95 to 99 % confident.

[00:24:22] We need to change the base site because this test was successful but it can also be, we should not change the base site because it was actually 95% statistically significant that this performs worse than the control. 

[00:24:36] So should you or should you not change the base site based off of set steak? That’s your answer there and then you want to communicate, you want to make sure that, hey, this positioning actually did perform better than the control. 

[00:24:48] What’s our next step? What does our sales enablement look like? Should we run a webinar based on this new positioning? 

[00:24:53] Should we update our decks with our new visuals because we just ran a test and we saw that these visuals work better than those. 

[00:25:00] What did you learn from your experiment that should be applied cross functionally that needs to happen so that everyone at the organization can really see how impactful CRO is to your business and then do it all over again. 

[00:25:14] When I was running conversion optimization at drift, I sat on the demand gen team and if we weren’t running experiments on the website, we weren’t hitting our demand goals. 

[00:25:23] CRO was an incredibly important motion for lead gen and overall conversion on the website. So, the fun thing I think about CRO is that you really are never done.

[00:25:35] Your job is never finished because you change the base site, you start again. So I shared a few examples of some experiments that we ran with Dandy but here are some other test examples that you can run at almost any level regardless of if you’re at level one or level four. 

[00:25:53] So campaign based experimentation or single page A/B testing, for example, you’re focusing on one specific page.

[00:26:01] You’re either driving traffic there from a campaign perspective or a paid perspective. And you want to really increase the overall conversion rate of that page from an A/B test perspective. 

[00:26:11] From a UX and UI improvements, you can change the content length. Do people want to read? When they come onto your website or do they just want short form bullet points?

[00:26:20] What does the mobile versus desktop experience look like? It doesn’t have to be one to one. Your mobile experience should be mobile optimized. 

[00:26:28] So should the content be shorter? Should the images be smaller? Should the images be fewer and far between? Can you add additional CTAs to the page? 

[00:26:37] Because people aren’t converting because your form is all the way down at the bottom and you have nothing that points to that point of conversion.

[00:26:43] CTA colors, as we’ve talked about but also typography. You probably have a main and secondary typeface that your company uses. What if you swapped them? 

[00:26:52] What if your secondary typeface was your primary? And how does your traffic respond to that. Relevancy experiments can be things like headline variations, hero image variations, CTA text variations, on page copy.

[00:27:04] We’ve kind of talked about this but when people are clicking in from an ad or clicking in from an email, what did they see in that? 

[00:27:12] In that source that brought them to the website and how can you make sure that post click experience is relevant, is mimicking what they saw before they came to your site that brought them to your site. Cross page experiencesor full site conversions.

[00:27:28] You can change the conversion experience. So, what does your form experience look like? How many fields do you display when people see your form for the first time? 

[00:27:37] Can you only show first name, last name, email and use progressive profiling? If that person’s already in your database, they hit submit and they’re good to go.

[00:27:44] But if you don’t know them, can you expose more fields on the front end so that you can get more information about them? 

[00:27:50] What does your nav bar experience look like? I’m sure you have a primary CTA in your nav bar but as you scroll, does it stick? Does it follow you as you scroll? And if it does what if you change the color?

[00:28:00] So it’s white when you scroll and then it changes to black and it simplifies. So you just have your primary CTA and then the next best page you want someone to go on. So there’s so much you can do with just your nav experience or copy on your form fields instead of full name it can say, what should we call you?

[00:28:16] Instead of email, can it say business email? Instead of just displaying a button that says click to get started, can you expose an email field that passes to like the larger form and makes it easier to fill out your conversion experience. 

[00:28:33] There’s so much that can be done there or you can add additional conversion points. So adding inline CTAs, when you have product call outs, that direct them to another page that talks more specifically about that product or an offer page surrounding that specific product, adding inline promotional CTAs. 

[00:28:50] So embedded CTAs that drive to content and then what should that content be? And how should that content be presented and what should the color look like and what should the CTA color look like?

[00:28:59] There’s tests that can be done within tests on your blog, do you have like a subscription pop up? And if you don’t, what if you add one? And then if you do, what if you change the background color or the background image or how you position your newsletter and what kind of social proof do people like?

[00:29:17] Do they like number of subscribers or calling out different verticals or calling out different business units, anything like that. 

[00:29:24] There’s so much that can be tested in your pop ups or if you have one CTA above the fold, what if you had two and what if you test which one is listed first and which one is listed second and then what are the colors and what are the texts so much that can be done for cross page conversions.

[00:29:42] These are more like customer journey conversion. So after you’re a hero going on to the page experiment experience, what is the next best element to have on the page? 

[00:29:50] Should it be a content offer and then social proof and then product information or social proof content offer product info or product info? Combination of all three of those. 

[00:29:59] What is the next best thing that someone wants to see when they scroll on your page to increase your conversion rate or increase page views per session or time on site or influence bounce rate or what is the next best offer that someone wants to see?

[00:30:14] So if there’s a returning visitor, what is the CTA copy? Should it say, convert now, convert this time. How should you change your experience for returning visitors? 

[00:30:27] What should your headline variations be for an enterprise visitor versus a mid market visitor? Or what is your page experience for a user who just attended a webinar versus that same page experience for a first time visitor?

[00:30:39] They shouldn’t be treated the same because two different things about them. So that kind of wraps up how we think about conversion rate optimization at Dandy. 

[00:30:49] Thank you so much for listening and tuning in and attending this conference. If you ever want to nerd out about conversion rate optimization or brainstorm your next test, I’m on Twitter at sara_pion or LinkedIn, Sarah pion.

[00:31:04] Hope you enjoy the rest of the sessions and have a good rest of your day.