Results

Humans vs AI

A/B testing OpenAI's GPT-3

In this friendly competition, we tested AI-generated copy against human-written copy for participating websites. Check out the results of the ongoing competition below.

1

👩 Human Copy Win

3

🤖 AI Copy Wins

3

👩 Human & 🤖 AI Ties

18

Confirmed participants

1

Awaiting Result

10

Inconclusive

Sorted by recency of results

Website

Test Type

Industry

Uplift %

Statistical Significance

Winner

Culligan Water

Website
Test Type

Headline

Industry

Consumer Services

Uplift %

24.3%

Statistical Significance

95%

Winner

🤖AI Copy

Test Duration:

👩 Human Copy
Human Copy 1
Culligan Human Copy
🤖 AI Copy
AI Variation 1
Culligan Ai Copy

The test was run by the brand’s agency partner: Optimized-Marketing.com

Clark Germany GmbH

Website
Test Type

Headline

Industry

Finance

Uplift %

15.77%

Statistical Significance

100%

Winner

🤖AI Copy

Test Duration:

👩 Human Copy
Human Copy 1
Clark Germany Gmbh Human Copy
🤖 AI Copy
AI Variation 1
Clark Germany Gmbh Ai Copy Variation 1 2x
AI Variation 2
Clark Germany Gmbh Ai Copy Variation 2 2x
AI Variation 3
Clark Germany Gmbh Ai Copy Variation 3 2x

All three AI copies won. Variation 2 resulted in the maximum uplift (15.77%) whereas Variation 1 and Variation 3 resulted in an uplift of 9.13% & 7.13% respectively with a statistical significance of >90%. 

Schneiders

Website
Test Type

Offer CTA

Industry

eCommerce

Uplift %

7.06%

Statistical Significance

91%

Winner

🤖AI Copy

👩 Human Copy
Human Copy 1
Schneiders Human Copy 2x
🤖 AI Copy
AI Variation 1
Schneiders Ai Copy Variation 1 2x

Booking.com

Website
Test Type

CTA

Industry

Travel

Uplift %

1.7%

Statistical Significance

95%

Winner

👩Human Copy

👩 Human Copy
Human Copy 1
Booking 2 Human Copy Variation 2
Human Copy 2
Booking 2 Human Copy Variation 1
🤖 AI Copy
AI Variation 1
Booking 2 Ai Copy

Human copy 1 won this test. We have shown blurred screenshots to mask the identity of the hotel property.

Springworks

Website
Test Type

Headline

Industry

SaaS

Uplift %

-

Statistical Significance

-

Winner

⚖️Human-AI Tie

Test Duration:

👩 Human Copy
Human Copy 1
Trivia Human Copy
🤖 AI Copy
AI Variation 1
Trivia Ai Copy

The uplift between human copy and AI copy was less than 5% (and statistically insignificant too). Hence for practical purposes, we conclude the test to be a tie.

Zinger Chairs

Website
Test Type

Headline

Industry

eCommerce

Uplift %

-

Statistical Significance

-

Winner

⚖️Human-AI Tie

Test Duration:

👩 Human Copy
Human Copy 1
Zinger Human Copy
🤖 AI Copy
AI Variation 1
Zinger Ai Copy 2x

The uplift between human copy and AI copy was less than 5% (and statistically insignificant too). Hence for practical purposes, we conclude the test to be a tie.

Prospects.ac.uk

Website
Test Type

Product Description

Industry

Education

Uplift %

-

Statistical Significance

-

Winner

⚖️Human-AI Tie

Test Duration:

👩 Human Copy
Human Copy 1
Prospects Human Copy 2x
🤖 AI Copy
AI Variation 1
Prospects Ai Copy Variation 1 2x
AI Variation 2
Prospects Ai Copy Variation 2 2x

The uplift between human copy and AI copy was less than 5% (and statistically insignificant too). Hence for practical purposes, we conclude the test to be a tie.

Booking.com

Website
Test Type

Page copy

Industry

Travel

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Booking 3 Human Copy
🤖 AI Copy
AI Variation 1
Booking 3 Ai Copy

The test has been deemed inconclusive since the result was not statistically significant i.e. the statistical significance threshold of 90%.

Booking.com

Website
Test Type

Headers

Industry

Travel

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Booking Human Copy
🤖 AI Copy
AI Variation 1
Booking Ai Copy

The test has been deemed inconclusive since the result was not statistically significant i.e. the statistical significance threshold of 90%.

Mogo

Website
Test Type

Headline

Industry

Finance

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Mogo Human Copy (1)
🤖 AI Copy
AI Variation 1
Mogo Ai Copy (1)

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

RV Trader

Website
Test Type

Headline

Industry

Travel & Hospitality

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Rv Trader Human Copy 2x Cmp
🤖 AI Copy
AI Variation 1
Rv Trader Ai Copy 2x Cmp

The test has been deemed inconclusive due to an error in the configuration of the test which led to erroneous results.

Eurosender

Website
Test Type

Headline

Industry

Logistics

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Eurosender Human Copy
🤖 AI Copy
AI Variation 1
Eurosender Ai Copy Variation 1
AI Variation 2
Eurosender Ai Copy Variation 2
AI Variation 3
Eurosender Ai Copy Variation 3

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

Soapbox

Website
Test Type

Headline

Industry

SaaS

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Soapbox Human Copy
🤖 AI Copy
AI Variation 1
Soapbox Ai Copy

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

Integrify

Website
Test Type

Headline

Industry

SaaS

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Integrify Human Copy Control
🤖 AI Copy
AI Variation 1
Integrify Ai Copy Variation 1
AI Variation 2
Integrify Ai Copy Variation 2

The participating website used the Multi-Armed Bandit (MAB) approach instead of A/B testing and hence we have considered it as inconclusive. However, the winning variation in the MAB test was generated by an AI.

Timeesci.com

Website
Test Type

Page copy

Industry

eCommerce

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Talleres De Ingenieria Human Copy
🤖 AI Copy
AI Variation 1
Talleres De Ingenieria Ai Copy

The test has been deemed inconclusive since the website had very low traffic to conclude the test on.

Adworld

Website
Test Type

CTA

Industry

Event

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

👩 Human Copy
Human Copy 1
Adworld Human Copy 2x
🤖 AI Copy
AI Variation 1
Adworld Ai Copy Variation 1 2x

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

CarGurus

Website
Test Type

Product Description

Industry

Automotive

Uplift %

-

Statistical Significance

-

Winner

Inconclusive

Test Duration:

👩 Human Copy
Human Copy 1
Cargurus Human Copy 2x
🤖 AI Copy
AI Variation 1
Cargurus Ai Copy Variation 1 2x

A bug was reported in the test where the copy was not appearing on pages as expected, which means that the AI-generated text wasn’t displayed to users who visited the test page. Hence, we have termed this test as inconclusive.

Website 1

Test Type

Headline

Industry

Marketplace

Uplift %

-

Statistical Significance

-

Winner

Awaiting Results

👩 Human Copy
Human Copy 1
Cardmarket Human Copy 2 Blurred
🤖 AI Copy
AI Variation 1
Cardmarket Ai Copy Variation 2 Blurred
AI Variation 2
Cardmarket Ai Copy Variation 1 Blurred

We have not shown the website name and clear screenshots for ongoing tests to avoid polluting the test data.
We will showcase them once we have statistically significant results.

Get notified as more results pour in

Simply leave your email below or follow us on @VWO to stay updated

Get more details on the first set of the test results- How AI-powered A/B testing is opening new doors for experimentation.

Request access to AI that recommends web copy

If you'd like to get early access to the AI-enabled feature shown above, submit your details and we will help you set up a VWO account with this feature enabled. You can then try this feature on your own website and set up A/B tests to measure its effectiveness

You don't need developer help for using this feature or for implementing A/B tests in VWO.

If you have any feedback for us or have a query, shoot us an email at support@vwo.com