Download Free: A\/B Testing Guide<\/em><\/span><\/a><\/h3>\n\n\n\nBut the typical suite of conversion optimization \u201cuser research\u201d tactics falls into a small set of familiar categories:<\/p>\n\n\n\n
\n- Use heatmaps<\/a>\/scrollmaps.<\/li>\n\n\n\n
- Survey users on site.<\/li>\n\n\n\n
- Watch session recordings<\/a>.<\/li>\n\n\n\n
- Do live user testing<\/a>.<\/li>\n<\/ul>\n\n\n\n
These are perfectly fine methods of learning about your user. This article, however, will focus on a treasure trove of user feedback that most ecommerce sites are underutilizing despite having abundant amounts of it: product reviews.<\/strong><\/p>\n\n\n\nProduct reviews give you a window into the psychology of the most qualified customers ever, those who purchased.<\/p>\n\n\n\n
You can learn:<\/p>\n\n\n\n
\n- What features or benefits do they value?<\/li>\n\n\n\n
- What features or benefits do they not value?<\/li>\n\n\n\n
- Why did they purchase?<\/li>\n\n\n\n
- What words and phrases do they use to describe the products and benefits?<\/li>\n<\/ul>\n\n\n\n
It\u2019s also free and fast; the reviews already exist on your site; and there is no need to pay for participants or wait to collect data.<\/p>\n\n\n\n
But the key to successfully using product reviews is in having a method to analyze them quantitatively. So you can extract actionable, numerical takeaways, not just vague high-level ideas.<\/p>\n\n\n\n
Here, we\u2019re going to tell a story about a quantitative method of analyzing open-ended feedback like product reviews. We call this method the Sort and Measure method<\/strong>. The information you can extract from it helps uncover which benefits are most valued by customers which can help shape copy and site design choices. The ideas you extract from applying this method can lead to better, more data-backed AB test hypotheses.<\/p>\n\n\nAnalyzing Product Reviews to Improve Product Page Copy<\/h2>\n\n\n
In this case study, we analyzed hundreds of product reviews for the site Amerisleep, a multimillion-dollar online mattress company in the United States.<\/p>\n\n\n\n![\"amerisleep__shop_eco-friendly__memory_foam_mattresses\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/Amerisleep__Shop_Eco-Friendly__Memory_Foam_Mattresses.png\")
<\/figure>\n\n\n\nUsing our observation, we were able to quantitatively rank how important different benefits were to the users\u2014comfort, pain, relative temperature regulation, environmental manufacturing, cost, returns, shipping, and more.<\/p>\n\n\n\n
Knowing how important these benefits were allowed us to craft detailed, long form product detail pages with confidence. Instead of guessing, we had real data that told us which benefits mattered and which did not.<\/p>\n\n\n\n
If you have a 7- or 8-figure ecommerce brand, it\u2019s possible that you can get the same results with this technique. Let\u2019s meet the company.<\/p>\n\n\n
Meet Amerisleep: An Online Mattress Leader with Multiple Product Benefits<\/h2>\n\n\n
Online mattresses are an ultra-competitive space right now. Amerisleep is a leading player in this space. It has a high-end mattress that it feels has several advantages:<\/p>\n\n\n\n
\n- (Comfort) Specially designed memory foam<\/li>\n\n\n\n
- (Options) Available in 5 different firmness levels<\/li>\n\n\n\n
- (Environmental) Manufactured in an environmentally friendly process<\/li>\n\n\n\n
- (Temperature) Designed to avoid overheating<\/li>\n\n\n\n
- (Cost) More affordable than retail memory foam mattresses<\/li>\n\n\n\n
- (Returns) They have a great \u201csleep trial\u201d policy<\/li>\n<\/ul>\n\n\n\n
Having lots of benefits sounds great, but it leads to unanticipated site design challenges.<\/p>\n\n\n
The Problem: Which Benefit to Emphasize?<\/h2>\n\n\n
Imagine you are the VP of Ecommerce for Amerisleep and also in charge of AB testing new product detail page copy to improve sales.<\/p>\n\n\n\n
It\u2019s nice to have a lot of product benefits, but it means that you have some predicaments when improving your product detail pages:<\/p>\n\n\n\n
\n- Which of these benefits should you emphasize?<\/li>\n\n\n\n
- If you list all of them, will it overwhelm the shopper?<\/li>\n\n\n\n
- Do shoppers even care about [insert benefit]?<\/li>\n\n\n\n
- Which benefits should you put at the top of the page, and which down below?<\/li>\n<\/ul>\n\n\n
How People Solve This: On-Site Surveys<\/h2>\n\n\n
How would you solve this problem and understand which benefits are more important than others?<\/p>\n\n\n\n
The most typical ecommerce user research method for extracting this information is to ask users in an on-site survey.\u00a0This is a useful method and can work. You use a Hotjar, Qualaroo, or VWO-style poll and ask shoppers which benefits they care about.<\/p>\n\n\n\n
\n- You could ask an open-ended question: What are you looking for in a mattress?<\/li>\n<\/ul>\n\n\n\n
![\"what-are-you-looking-for-in-a-mattress\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/What-are-you-looking-for-in-a-mattress.jpg\")
<\/figure>\n<\/div>\n\n\n\n- You could ask a multiple-choice question: Which of these benefits of a mattress is most important to you?<\/li>\n<\/ul>\n\n\n\n
This is a solid method of starting to understand your users. If you do this consistently, in our experience, you\u2019d be ahead of a lot of competitors.<\/p>\n\n\n\n
But this method has a key drawback\u2014you don\u2019t know how qualified the respondents are.<\/strong><\/p>\n\n\n\nThis is especially an issue for high-end brands. What if you get lot of responses about the price being too high, but the respondents weren\u2019t your target customers and weren\u2019t ever going to buy from you anyway?<\/p>\n\n\n\n
This is where product reviews become extremely useful.<\/p>\n\n\n
Product Reviews: You Already Have the Answers on Your Site<\/h2>\n\n\n
A great way to solve this problem is to survey only users who have bought the mattress. You can do this on the success page by asking: What were the key reasons you bought your mattress from us today? But the vast majority of successful ecommerce brands already have these survey results on their sites: these are in the form of product reviews.<\/p>\n\n\n\n
Product reviews are exactly what benefits users consider important: it\u2019s a survey of only customers who bought, and it\u2019s open-ended and in their own words.<\/strong><\/p>\n\n\n\nNote: The case study in this article used a buyer verification process for leaving reviews. If such a process doesn\u2019t exist, it is possible that non-buyers could leave reviews. But in our experience, most stores have too few reviews, not too many.<\/em><\/p>\n\n\n\nReviews also have a tendency of being brutally honest (something many store owners know too well), as many will include criticism of what they didn\u2019t like and what could be improved, even though they bought anyways.<\/p>\n\n\n\n
Download Free: A\/B Testing Guide<\/em><\/span><\/a><\/h3>\n\n\nSort and Measure Method: Extract Quantitative Answers to Make Decisions<\/h2>\n\n\n
But how are you going to turn product reviews into actionable information to inform your product page redesign<\/a> or any other AB test on your site?<\/p>\n\n\n\nEnter the Sort and Measure Method. Here\u2019s what it is, in a nutshell.<\/p>\n\n\n
Casual Browsing: A Recipe for Personal Biases<\/h3>\n\n\n
Typically, what happens with survey data at an ecommerce company is that it\u2019s emailed around and discussed a bit, and people then interpret it how they want.<\/p>\n\n\n\n
One person says: \u201cI told you, most of the reviewers were talking about back pain relief!\u201d<\/em><\/p>\n\n\n\nAnother employee says: \u201cWell, what really struck me were the ones saying our environmental manufacturing was important.\u201d<\/em><\/p>\n\n\n\nThe debate can be never-ending. Most companies end up going with decisions based on either who is the loudest or who is the most senior.<\/p>\n\n\n\n
For example, even before we used this technique, we did a large redesign and rewrite of their product detail pages. Using VWO Testing<\/a> to A\/B test the redesign, we got a great, 14% lift in checkouts – a lift that\u2019s worth 7 figures annually.<\/p>\n\n\n\nAlthough this was a great result, we had email exchanges like this between me and the copywriting consultant we partnered with, debating where to put emphasis. Here is an email I wrote to him about a certain benefit I felt wasn\u2019t that important:<\/p>\n\n\n
These are perfectly fine methods of learning about your user. This article, however, will focus on a treasure trove of user feedback that most ecommerce sites are underutilizing despite having abundant amounts of it: product reviews.<\/strong><\/p>\n\n\n\n Product reviews give you a window into the psychology of the most qualified customers ever, those who purchased.<\/p>\n\n\n\n You can learn:<\/p>\n\n\n\n It\u2019s also free and fast; the reviews already exist on your site; and there is no need to pay for participants or wait to collect data.<\/p>\n\n\n\n But the key to successfully using product reviews is in having a method to analyze them quantitatively. So you can extract actionable, numerical takeaways, not just vague high-level ideas.<\/p>\n\n\n\n Here, we\u2019re going to tell a story about a quantitative method of analyzing open-ended feedback like product reviews. We call this method the Sort and Measure method<\/strong>. The information you can extract from it helps uncover which benefits are most valued by customers which can help shape copy and site design choices. The ideas you extract from applying this method can lead to better, more data-backed AB test hypotheses.<\/p>\n\n\n In this case study, we analyzed hundreds of product reviews for the site Amerisleep, a multimillion-dollar online mattress company in the United States.<\/p>\n\n\n\n Using our observation, we were able to quantitatively rank how important different benefits were to the users\u2014comfort, pain, relative temperature regulation, environmental manufacturing, cost, returns, shipping, and more.<\/p>\n\n\n\n Knowing how important these benefits were allowed us to craft detailed, long form product detail pages with confidence. Instead of guessing, we had real data that told us which benefits mattered and which did not.<\/p>\n\n\n\n If you have a 7- or 8-figure ecommerce brand, it\u2019s possible that you can get the same results with this technique. Let\u2019s meet the company.<\/p>\n\n\n Online mattresses are an ultra-competitive space right now. Amerisleep is a leading player in this space. It has a high-end mattress that it feels has several advantages:<\/p>\n\n\n\n Having lots of benefits sounds great, but it leads to unanticipated site design challenges.<\/p>\n\n\n Imagine you are the VP of Ecommerce for Amerisleep and also in charge of AB testing new product detail page copy to improve sales.<\/p>\n\n\n\n It\u2019s nice to have a lot of product benefits, but it means that you have some predicaments when improving your product detail pages:<\/p>\n\n\n\n How would you solve this problem and understand which benefits are more important than others?<\/p>\n\n\n\n The most typical ecommerce user research method for extracting this information is to ask users in an on-site survey.\u00a0This is a useful method and can work. You use a Hotjar, Qualaroo, or VWO-style poll and ask shoppers which benefits they care about.<\/p>\n\n\n\n This is a solid method of starting to understand your users. If you do this consistently, in our experience, you\u2019d be ahead of a lot of competitors.<\/p>\n\n\n\n But this method has a key drawback\u2014you don\u2019t know how qualified the respondents are.<\/strong><\/p>\n\n\n\n This is especially an issue for high-end brands. What if you get lot of responses about the price being too high, but the respondents weren\u2019t your target customers and weren\u2019t ever going to buy from you anyway?<\/p>\n\n\n\n This is where product reviews become extremely useful.<\/p>\n\n\n A great way to solve this problem is to survey only users who have bought the mattress. You can do this on the success page by asking: What were the key reasons you bought your mattress from us today? But the vast majority of successful ecommerce brands already have these survey results on their sites: these are in the form of product reviews.<\/p>\n\n\n\n Product reviews are exactly what benefits users consider important: it\u2019s a survey of only customers who bought, and it\u2019s open-ended and in their own words.<\/strong><\/p>\n\n\n\n Note: The case study in this article used a buyer verification process for leaving reviews. If such a process doesn\u2019t exist, it is possible that non-buyers could leave reviews. But in our experience, most stores have too few reviews, not too many.<\/em><\/p>\n\n\n\n Reviews also have a tendency of being brutally honest (something many store owners know too well), as many will include criticism of what they didn\u2019t like and what could be improved, even though they bought anyways.<\/p>\n\n\n\n But how are you going to turn product reviews into actionable information to inform your product page redesign<\/a> or any other AB test on your site?<\/p>\n\n\n\n Enter the Sort and Measure Method. Here\u2019s what it is, in a nutshell.<\/p>\n\n\n Typically, what happens with survey data at an ecommerce company is that it\u2019s emailed around and discussed a bit, and people then interpret it how they want.<\/p>\n\n\n\n One person says: \u201cI told you, most of the reviewers were talking about back pain relief!\u201d<\/em><\/p>\n\n\n\n Another employee says: \u201cWell, what really struck me were the ones saying our environmental manufacturing was important.\u201d<\/em><\/p>\n\n\n\n The debate can be never-ending. Most companies end up going with decisions based on either who is the loudest or who is the most senior.<\/p>\n\n\n\n For example, even before we used this technique, we did a large redesign and rewrite of their product detail pages. Using VWO Testing<\/a> to A\/B test the redesign, we got a great, 14% lift in checkouts – a lift that\u2019s worth 7 figures annually.<\/p>\n\n\n\n Although this was a great result, we had email exchanges like this between me and the copywriting consultant we partnered with, debating where to put emphasis. Here is an email I wrote to him about a certain benefit I felt wasn\u2019t that important:<\/p>\n\n\n\n
Analyzing Product Reviews to Improve Product Page Copy<\/h2>\n\n\n
<\/figure>\n\n\n\nMeet Amerisleep: An Online Mattress Leader with Multiple Product Benefits<\/h2>\n\n\n
\n
The Problem: Which Benefit to Emphasize?<\/h2>\n\n\n
\n
How People Solve This: On-Site Surveys<\/h2>\n\n\n
\n
<\/figure>\n<\/div>\n\n\n\n
Product Reviews: You Already Have the Answers on Your Site<\/h2>\n\n\n
Download Free: A\/B Testing Guide<\/em><\/span><\/a><\/h3>\n\n\n
Sort and Measure Method: Extract Quantitative Answers to Make Decisions<\/h2>\n\n\n
Casual Browsing: A Recipe for Personal Biases<\/h3>\n\n\n
![\"img_6600_png\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/IMG_6600_PNG.jpg\")
<\/figure>\n<\/div>\n\n\n![\"brian_response-min\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/Brian_response-min.jpg\")
<\/figure>\n<\/div>\n\n\n![\"testimonials\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/Testimonials.jpg\")
<\/figure>\n<\/div>\n\n\n![\"sort_and_measure_2-min\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/sort_and_measure_2-min.jpg\")
<\/figure>\n\n\n\n![\"amerisleep_user_research_results\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/Amerisleep_user_research_results.jpg\")
<\/figure>\n\n\n\n![\"amerisleep_user_research_results-1\"](\"https:\/\/static.wingify.com\/vwo\/uploads\/sites\/3\/2017\/07\/Amerisleep_user_research_results-1.jpg\")
<\/figure>\n\n\n\n