{"id":54957,"date":"2020-11-04T15:21:02","date_gmt":"2020-11-04T09:51:02","guid":{"rendered":"https:\/\/vwo.com\/blog\/?p=54957"},"modified":"2025-05-05T13:36:49","modified_gmt":"2025-05-05T08:06:49","slug":"ab-testing-gpt-3-5-turbo-ai","status":"publish","type":"post","link":"https:\/\/vwo.com\/blog\/ab-testing-gpt-3-5-turbo-ai\/","title":{"rendered":"A\/B Testing Website Copy With GPT-3.5 Turbo Opening New Doors for Experimentation Using AI"},"content":{"rendered":"\n

From initially helping humans out with redundant and manual tasks, to now mastering creative jobs such as making original art or composing music, AI has evolved and transformed in unprecedented ways. One such creative job that bots are surprisingly good at is writing copy! Yes, GPT-3.5 Turbo(Generative Pre-trained Transformer 3) is a neural-network powered AI that can produce nearly flawless text relevant to the given context. Built by OpenAI<\/a>, a San Francisco-based research lab, GPT-3.5 Turbo is a third-generation powerful language generator that uses machine learning to predict and produce text, almost<\/em> like a human.\u00a0<\/p>\n\n\n\n

Download Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\n\n

If you expand on GPT-3.5 Turbo, here\u2019s what it denotes: <\/p>\n\n\n\n

Generative:<\/strong> Indicating that the goal of the model is to generate text by predicting one word at a time in a given sentence. <\/p>\n\n\n\n

Pre-trained:<\/strong> Indicating that a huge amount of data has been fed into the system to train it.<\/p>\n\n\n\n

Transformer:<\/strong> Indicating the algorithm used by the AI model, which specializes in natural language processing, i.e., how words are used in a language and what they mean. <\/p>\n\n\n

\n
\"AB<\/figure>\n<\/div>\n\n\n

Once GPT-3.5 Turbo is fed a prompt, it generates streams of text by predicting the possibility of a sentence existing in this world. Currently, the functionality is in beta and only offered to a select group (including VWO<\/a>) through an API accessible via the cloud. <\/p>\n\n\n\n

Let’s face it – copywriting is no easy task. GPT-3.5 Turbo’s robust and flexible language model can produce a short copy at scale. If you add to that the ability to test copy versions, you can get the best of both worlds. Also, some of the most commonly run tests revolve around webpage copy. So, integrating Open AI’s GPT-3.5 Turbo API with VWO Testing<\/a> was the most natural and logical thing for us to do.<\/p>\n\n\n\n

Our new feature enables you to use AI-generated copy to create variations for your website copy and deploy them without any help from IT. You can also test the AI-generated copy against the original human-written copy on your website. The next section covers how popular brands uncovered the practical implications of our new feature via a friendly contest between human-written and AI-generated copy. <\/p>\n\n\n

VWO\u2019s Human vs. AI competition<\/h2>\n\n\n

In August this year, VWO hosted a friendly competition<\/a> between copy written by human copywriters and that by our new feature powered by OpenAI’s GPT-3.5 Turbo API. We invited participants from all over the world and tested AI-generated text against human-written one for their webpages with sufficient traffic via VWO or any other testing platform<\/a> they were using. <\/p>\n\n\n\n

Over 450 brands were given access to the AI copy generating feature during the course of this competition. Among the 18 shortlisted participants were Booking.com, Clark Germany GmbH, and Schneiders, to name a few. The AI feature was able to generate copies in various languages such as Spanish, German, Portuguese, etc. The participants were highly satisfied with the accuracy of the output in these languages.<\/p>\n\n\n\n

All participants had to set up their tests keeping the original website copy as the control and the AI-generated one(s) as the variation. <\/p>\n\n\n\n

Results of the competition<\/h3>\n\n\n\n

Among the 18 tests run by the confirmed participants, 1 had an existing (or new) human written copy as the winner, 3 had AI copy as the winner, 3 were declared as a tie, 2 are still awaiting results, and 9 were inconclusive. <\/p>\n\n\n\n

Let\u2019s take a look at some of the tests where the AI-generated copy won<\/strong>:<\/p>\n\n\n\n

Schneiders [An eCommerce store for horse wear & equipment]<\/h4>\n\n\n\n

The team tested their topmost banner copy by creating a variation of the original page using the AI-backed language generator. Here\u2019s a look at the control and variation from the test:<\/p>\n\n\n\n

\"Human
Control [Human-written copy]<\/em><\/figcaption><\/figure>\n\n\n\n
\"Gpt
Variation [AI-generated copy<\/em>]<\/figcaption><\/figure>\n\n\n\n

Once statistically significant results were achieved, the A\/B test<\/a> declared the variation to be the winner as it led to a 7.06% uplift in their banner clicks.<\/p>\n\n\n\n

Clark Germany GmbH [An insurance agency based out of Frankfurt]<\/h4>\n\n\n\n

3 variations of the page headline were created using the AI copy and pit against the control. The test was run for 48 days. Following are the control and variations of the test: <\/p>\n\n\n\n

\"Human
Control [Human-written copy]<\/em>
English translation: Manage Your Insurance Digitally<\/em>
<\/figcaption><\/figure>\n\n\n\n
\"Gpt
Variation 1 [AI copy]<\/em>
English Translation: Manage Your Insurance Digitally<\/em>
<\/figcaption><\/figure>\n\n\n\n
\"Variation
Variation 2 [AI copy]<\/em>
English Translation: Keep Track Of Your Insurances<\/em>
<\/figcaption><\/figure>\n\n\n\n
\"Variation
Variation 3 [AI copy]<\/em>
English Translation: Clark Is Your Digital Insurance Manager<\/em>
<\/figcaption><\/figure>\n\n\n\n

Once the test reached conclusion (statistical significance >  90%), all 3 variations outperformed the control. Variation 2 resulted in the maximum uplift in their CTA clicks (15.77%), while Variation 1 and 3 resulted in an uplift of 9.13% & 7.13%, respectively.<\/p>\n\n\n\n

Here\u2019s the test that declared the human copy as the winner<\/strong>:<\/p>\n\n\n\n

Booking.com [A global travel company] <\/h4>\n\n\n\n

The team at Booking.com tested<\/a> the CTA on their hotel booking pages. 2 human-written copies were pitted against an AI-generated one. Following are the variations they created:<\/p>\n\n\n\n

\"Human
Variation 1 [Human-written copy #1<\/em>]<\/figcaption><\/figure>\n\n\n\n
\"Gpt
Variation 2 [Human-written copy #2]<\/em><\/figcaption><\/figure>\n\n\n\n
\"Variation
Variation 3 [AI-generated copy]<\/em><\/figcaption><\/figure>\n\n\n\n

The human copy #1 won the test as it resulted in a 1.7% uplift in the CTA conversion rate.<\/p>\n\n\n\n

Here\u2019s a test that resulted in a human-AI tie<\/strong>:<\/p>\n\n\n\n

Springworks [A SaaS company based out of India]<\/h4>\n\n\n\n

The team at Springworks tested their landing page headline<\/a> by creating a variation using the AI-generated copy and pitting it against the original (control). Their goal was to improve clicks on the \u2018Add Trivia\u2019 CTA. The test was run for 8 days. Here\u2019s a look at the control and variation:<\/p>\n\n\n

\n
\"Human
Control [Human-written copy]<\/em>
<\/figcaption><\/figure>\n<\/div>\n\n
\n
\"Gpt
Variation [AI copy]<\/em><\/figcaption><\/figure>\n<\/div>\n\n\n

Since the difference in the uplift in CTA clicks between the control and variation was less than 5%, and the test results were statistically insignificant, the test was declared to be a tie. <\/p>\n\n\n\n

Download Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\n\n

Let\u2019s deep dive into the nitty-gritty of how VWO Testing<\/a> and GPT-3.5 Turbo work together.<\/p>\n\n\n

VWO Testing & GPT-3.5 Turbo<\/h2>\n\n\n

We integrated Open AI\u2019s GPT-3.5 Turbo API with our Visual Editor<\/a> so that every time you decide to run a test or deploy a change, you can generate copy recommendations that you can choose to create variations out of. This means you get to cut down on time spent brainstorming on variations and alternatives by having a library of AI-generated ideas readily available at your disposal. <\/p>\n\n\n\n

Whether you are looking to optimize headlines, CTA text, product descriptions, or any other text on your site, you can quickly generate alternatives and either directly deploy them or test them against your original copy, both without any developer help. Either way, by automating this aspect of experimentation<\/a>, you get to make your CRO program<\/a> more efficient and agile. <\/p>\n\n\n\n

Once you open VWO\u2019s Visual Editor<\/a> and click on any piece of text, you will find a \u2018Suggest Variations\u2019 option in the drop-down menu. Clicking on it will display a bunch of AI-powered copy suggestions (based on the existing copy) that you can choose from.<\/p>\n\n\n\n

\n