This website works best with JavaScript enabled. Learn how to enable JavaScript.

VWO Blog

A Blog Around Everything Conversion Optimization And Beyond

Never miss an update from VWO

A value for this field is required.

A/B testing browser extensions: do annoying notifications cause uninstallations?

Editor’s note: this is a case study written by one of our customers (InvisibleHand). Usually A/B testing is limited to web pages but they did an interesting A/B test which actually worked inside a browser extension. It’s one of a kind A/B testing case study and we are thankful to them for sharing it. So, go ahead and enjoy the case study!

Most people refer to web pages when they talk about A/B testing. However, A/B testing can be useful in other domains too. At InvisibleHand, we use Visual Website Optimiser to test the performance of different versions of a browser extension.

InvisibleHand is a browser extension that automatically compares prices for you. For example, when you browse a product at, InvisibleHand checks other retailers where this product is available, compares real-time prices and shows a notification like “Save 10%! This product is $10 cheaper ($89.99) at”, giving you a direct link to the product at a competing retailer. Since the launch in 2009 InvisibleHand enjoyed almost 2m downloads. We make money by earning commission on affiliate sales, so our revenues depend on the quality and quantity of our notifications.

A/B testing inside browser extension using IFrames

In order to make InvisibleHand available on all major browser platforms (Chrome, Firefox, Safari, IE), we try to reuse as much code as possible. In particular, the notification shown is injected as an iframe (as opposed to a div element, for example, or the built-in browser API) to enable us to share the same code across browsers and to control the behaviour of the extension from the server-side without having to ask users to install a new version every time we try a new feature. Crucially, this also enables us to perform A/B testing of new functionality.

Normal notifications in InvisibleHand

The iframe has significant control over the look of the notification. In particular, it can choose whether to be shown, which comes really handy if we decide to enable or disable certain kinds of notifications. The mechanism is simple:

  • The extension injects an invisible empty notification and sends a message with notification contents using postMessage()
  • The iframe analyses its contents and decides what to do
  • The iframe sends a message back to the notification telling it to make the iframe visible if necessary

Since the iframe is just a web-page, we can use Visual Website Optimiser to tests different variations of the extension. For example, the very first test we performed measured the likelihood of a click on a notification using two different fonts. However, what we really are interested in is what makes our users to uninstall InvisibleHand and how to prevent it.

Hypothesis: do annoying notifications cause uninstallations?

Our first hypothesis was that some of our notifications are more annoying than the others and they prompt the users to remove the extensions. In particular, we show money-saving notifications on google and other search engines when product searches are performed but these notifications are not highly accurate since the users are not always searching for a product and even when they are, they may not be interested in buying it. For example, a user may be researching a camera online and InvisibleHand notifications would be just a distraction if they don’t plan on buying it right now.

Annoying search notifications in InvisibleHand

A/B test implementation

So, we designed a variation of the test that doesn’t show notifications on search engines. In our iframe we have a check:

    var overloading = true;
    if (typeof shouldShowNotification == 'function') {
      overloading = shouldShowNotification(this);
    if (eventSource && overloading) {
      eventSource.postMessage(JSON.stringify({message: 'showNotification'}), '*');

This piece of code shows all notifications by default unless there is a function shouldShowNotification() defined that can keep a notification hidden. So we created a variation for A/B testing that defines this function:

   function shouldShowNotification(notification) {
      return notification.category != ‘search_engine’;

In this variation all notifications on all search engines would remain hidden even though they would still be injected into the page.

I mentioned earlier that InvisibleHand works with all major browsers but we usually introduce experimental features to only one platform, Chrome, to avoid the hassle of deployment and rollbacks. So, to limit the scope of the test to only one browser, we check the user agent:

    var browserEnabled = navigator.userAgent.toLowerCase().indexOf('chrome') > -1;
    if (browserEnabled) {
      var _vis_opt_account_id = 7013;
      //... the rest of the code

So, the VWO code isn’t executed on all other browsers. Cool!

Measuring uninstallations!

Measuring extension uninstallations on Google Chrome turned out to be slightly trickier. Since the uninstallation of a browser extension may happen when no notifications are shown and VWO code is not running, we decided to show a “goodbye” notification when the extension was uninstalled. The “goodbye” notification is hosted on the same domain as the normal notification; its primary purpose is to trigger a custom conversion goal:

    _vis_opt_register_conversion(1, 5);

A/B test results

It’s ironic that Visual Website Optimiser assumes that we want to maximise all conversion goals, so when the uninstallation rate was going up it was reported as an “improvement” [Editor’s note: great point! We will work on it]. Other than that, it worked just as expected, allowing us to measure the correlation between annoying notifications being disabled and the probability of uninstallation. The conclusion was that there is indeed a slight increase in the number of uninstallations when the notifications on search engines are shown but since the engagement rate (number of clicks) on these notification is fairly high, we concluded that the users would probably be better off with them rather than without.

One of the reasons we decided to move part of the functionality of InvisibleHand into an iframe is the ability to perform A/B tests and we’re quite happy with Visual Web Optimizer helping us to do this. As it turns out, it’s perfectly possible to A/B test browser extensions, as long as some parts of them are normal web pages.

Founder and Chairman of Wingify.

Comments (5)

Leave a Comment
  1. A/B testing, split testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.

    Significant improvements can be seen through testing elements like copy text, layouts, images and colors. However, not all elements produce the same improvements, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest improvements.

    Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. The test, in order to be effective, must reach an audience of a sufficient size that there is a reasonable chance of detecting a meaningful difference between the control and other tactics: see Statistical power.


Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login