on Conversion Rate Optimization

What You Should Know About the Optimizely “Vulnerability”

Update 25th June 2014 7pm IST: With the new version of VWO, we have now given an option to mask experiment names and variation names.

Update 5th June 2014 11 am IST: It looks like Optimizely is planning to fix the issues that have been identified – particularly they will allow exclusion of paused and draft tests from test settings. We’re glad they responded quickly to this. Gaining trust of businesses by fixing such issues will be good for A/B testing industry at large, and that was our original intention.

In the comments below, people mention that knowing what tests are running is possible using VWO as well. As we write in the article, it’s by definition possible for all A/B testing tools. What is not possible from the VWO test settings code right now is to know what tests are planned or what tests were conducted in the past (and hence give away business strategy). We believe this was an important issue and that’s what we highlighted in the post. Like Optimizely, for integration with analytics products, VWO shows variation names and experiment names. We’re soon going to release an option to replace that with account id and experiment id.

** End of Update **
With Visual Website Optimizer being a prominent vendor in the A/B testing industry, it’s our duty to comment and elaborate on popular concerns and queries. After reading this post, if you find that we’re wrong about anything, have misinterpreted something, or if you know of better approaches, we invite corrections (and we will promptly update the article). A/B testers are a tight-knit group who are passionate (like us) about providing marketers and web developers the tools to best the competition. This is why we feel so passionately about addressing these issues, because they may lead to the opposite effect.

Media outlets and the world at large recently found out what we’ve known for a long time: running A/B tests using Optimizely is not safe from a competitive point of view. Take a look at the following article:

Optimizely vulnerability lets you see what other sites are testing (Published on June 02, 2014)


To quote VentureBeat:

“The leak, which is embedded in the JavaScript code used to implement Optimizely on each site, doesn’t appear to endanger the sites or their fundamental security. But it does enable anyone, with very little effort, to see exactly what tests an Optimizely customer is conducting.”

The market at large is worried about this, and they should be. And we want to clarify why such vulnerabilities are unique to Optimizely’s approach and also expand on what makes Visual Website Optimizer (VWO) immune to them.

What Exactly are the “Vulnerabilities” and Why are They There?

Optimizely depends on external content delivery networks such as Akamai. There is a reason the delivery infrastructure has the word “content” in it. Akamai and other CDNs are primarily used to deliver static content like videos, images and plain text. Although CDNs are becoming better and more flexible by the day, the fact is that they were never meant to deliver content which is dynamically retrieved (like your Facebook feed or Gmail inbox). There are nuances to this but dynamic querying capabilities of a CDN are still very limited as compared to a database like MySQL.

CDNs have multiple servers across the world and they simply replicate these static content pieces across all servers. When someone visits a website using a CDN, it typically selects the geographically nearest server to the user and uses that to load the website content.

So What’s the Problem?

All A/B testing platforms need to deliver campaign/test settings in a dynamic manner specific to the URL a visitor is on. The problem with Optimizely’s approach is that they use a CDN meant for static content to serve their Javascript file that makes the changes in an A/B test. Since it’s a static file, Optimizely’s code contains ALL past, present and work-in-progress campaigns setup across the account, running or not running. Once this file is downloaded, it checks what URL the visitor is on and then applies the relevant A/B test changes to it.

Since test settings are delivered as JavaScript code, anyone can simply “View Source” in the browser to see what’s in it. Test settings being visible to anyone in the browser is not unique to Optimizely, it’s the standard method for most A/B testing tools that integrate via JavaScript. However, what’s unique with Optimizely is that you get to see not only the tests and campaigns for a specific page (say for but for all other pages on the website as well (say and other pages that aren’t even linked from the main website). Moreover, you also get to see all un-archived tests in Optimizely – old, present and even the ones you’re currently working on.

How is VWO Different?

We built our own home-grown CDN network that can serve dynamic content. The VWO approach is to dynamically generate page-specific settings for running campaigns only (not stopped or work-in-progress) with every page visit. For example, if visitors arrive on they will only be served those tests that are running on the new VWO’s early access page, nothing else.

Example output of VWO test campaign settings

VWO code

What’s Contained in the Publicly Accessible Optimizely Campaign Settings File?

Everything! That’s concerning. As at the time of writing this post, their publicly viewable campaign setting file contained:

  • All the A/B test and other optimization campaigns you are currently running across your entire website (even on the pages you wanted to keep secret from competitors)
  • All the campaigns you have run in the past that you haven’t archived
  • All the campaigns you are planning (which are in drafting stage)

Someone actually made a dedicated website for you to take a peek at other website testing and optimization campaigns. Take a look at Simply enter a URL of your competitor / target website and instantly know what campaigns they’re running and what ideas they’re planning to test in future.

Know what CNN is testing using Optimizely


Why Should You Care?

In essence, if you are using Optimizely (or any other vendor that takes the similar route of relying on external CDNs to deliver test settings):

  • All the URLs of any landing page that you are testing will be public even if you don’t link them from anywhere on the main website and even if you never wanted URLs to be seen by anyone else than who it was intended for.
  • If your competitors know the last 10 campaigns you ran, they can predict the direction the business is heading. This is almost like they knowing what is being discussed in internal meetings.
  • The new ideas you’re toying with will be public before you wanted them to be. They can know if you’re planning to test pricing or new offers and can react before you even had a chance to implement them.
  • All old and discarded attempts will be there for your competition and technically savvy users to see. So, if you were testing $49 v/s $69 for a product and you retained $69, people will know that the same product was available for cheaper one week back. And in the worst case, even if you have stopped the campaign, people can still see and use the old versions that you no longer wanted to be seen.

All in all, Optimizely users are currently exposed because competitors can go through their tests and get a sense of the direction the business is taking. Marketers all over are evidently shocked and concerned about Optimizely’s approach.


Another consequence of Optimizely’s approach: large download sizes and potentially slow website load times

If you dump all the campaigns and settings of all the tests across the website, old, current and future ones, the download size is bound to increase. People notice such stuff:


And since Optimizely’s integration is “one line” and synchronous, download of this hundreds of kilobytes (sometimes which goes into MBs) of data takes time. And it’s quite obvious that it directly impacts sales and conversions on eCommerce and other websites. Sample a few recent tweets:


(The last time we had a complaining tweet about VWO being slow was in 2012, when we had synchronous code. Now we have asynchronous code snippet for integration and our own blazing fast CDN.)

Granted that the static file delivered through an external CDN is cached and will typically be only downloaded only once per session (when a visitor visits the website). However, first experiences matter and if the website is slow even once, conversions, sales and revenue take a hit.

Like we said previously, VWO dynamically generates page-specific settings for running campaigns. This means that settings are not cached, but if you compare the response times of both approaches, you will see that they’re similar. (See a public comparison between response time of our approach v/s external static CDN approach.)

So, even without caching, our method of delivering test settings is not slow and obviously it has an advantage of not leaking your entire testing strategy.

How Visual Website Optimizer is Safe from These “Vulnerabilities”?

As briefly mentioned above, we have an entirely different architecture. There are couple of major differences in how tests are delivered and executed with VWO.

1. We have our own Content Delivery Network to allow for dynamic delivery

Instead of relying on an external CDN for delivering test and campaign settings, we have our own geographically distributed CDN which we keep expanding by adding new server locations. This enables us to deliver test settings specific to the URL a visitor is on.

2. With VWO, nobody can see what other pages you are running test and campaigns on

(unless they know the URLs of the pages you are running tests on).

Plus, our custom CDN is lightning fast, delivering settings quickly to test participants from across the world. Check for yourself on an independent monitoring system such as WatchMouse or Site24x7.

3. Our code is not “one-line” but is smart and configurable asynchronous code

VWO smart code looks like this:


The reason it “looks” complex is because it is smart. Unlike other A/B testing vendors’ integration codes where they first download test settings and then load rest of the page, this code downloads test settings parallel to the website.


The VWO code also has user configurable timeouts. So you can configure VWO for specific situations, like if test settings are not downloaded in one second, just timeout and show the original website. On the contrary, synchronous code in worst case hangs for 30 seconds in most browsers.

In an age where even a second of delay causes 7% loss in conversions, your business will suffer massively if it happens to stall for 30 seconds!

As an added feature, Visual Website Optimizer also offers self-hosted test settings where you can download and host test settings from your own server, removing all dependencies (except for logging) on any 3rd party servers. Many of our financial and banking customers use this.

VWO asynchronous code still has all the advantages of one time integration (you just have to copy-paste this code once on your website and you’re done). So apart from aesthetic value of the “one-line” integration to developers, there’s no real reason why you should prefer that instead of having code that allows for greater flexibility.

Bottomline: A/B testing is important, but so is the issue of leaking sensitive information to competitors and having a really slow website

We know that A/B testing is just so crucial to businesses and the adoption of A/B testing has been exploding for last many years. A/B testing and Conversion Rate Optimization (CRO) are here to stay. However, you should be prudent in making sure you don’t inadvertently expose information that you don’t want your competitors to know.

If you have any questions regarding the technology, architecture or security of VWO or any other vendor, please reach out to us. Simply leave a comment here or email, and we are happy to answer. If you want to talk to our sales, reach out to us on and we can advise you on how you can easily port your campaigns, reports and data from your existing vendor to VWO.

To make A/B testing earn trust of businesses at larger, we sincerely hope the right architecture and approach is adopted by all vendors. In fact, our engineering team is ready and willing to help other vendors fix the issues. If this issue is fixed, everybody wins.

Again, if we’re wrong anywhere in the article or misinterpreted anything or if you know of better approaches, we invite corrections (and we will promptly update the article).

Update: As we were about to publish this post, Optimizely published an update on their blog on how they’re going to allow their customers to mask experiment names. But they don’t talk about the real issue of all planned, future and past campaigns (along with URLs) being discoverable by competitors and how the same approach leads to a large, bloated settings file that could slow down a website.

Founder and CEO of Wingify.

Comments (32)

Leave a Comment
  1. Not sure this is correct: “And since Optimizely’s integration is “one line” and synchronous”

    From this page: “The Optimizely JavaScript file download occurs in parallel alongside other JavaScript and CSS files..”

    Seems to indicate async loading?

    Other than that, I agree with your points – the increase to file size (even if it is done in parallel, most users don’t have unlimited bandwidth..), and ability to view all experiments / changes, even those you weren’t bucketed in, is troublesome.

    1. All the JS and CSS files are downloaded by the browsers in parallel but the page loading is delayed by any JS/CSS file which is present on the page. With single line of code it means that if the resource is unavailable or takes a lot of time to respond, it will delay the page load, which is not the case with Async code.

  2. We used to use VWO and switched to Optimizely (not my decision) but I’ve been thinking of pushing for going back to the (new) VWO. But, we load the Optimizely script via a tag management system, so the Optimizely code doesn’t actually show up in our pages’ source code. Does that mean we are exempt from this vulnerability? Can anyone tell me definitively? Thanks.

  3. I wanna say ‘Daaaamn!’ and just leave it at that, but I can’t. You guys pulled some punches.

    When I was looking for an A/B testing tool, it came down to you guys and them. Here’s what I found.

    1. Optimizely is expensive. Just about twice as much.

    2. Optimizely uses a “project” system that limits the number of sites/clients you can test. Agencies like mine can’t really use Optimizely without forking over hundreds of dollars a month.

    3. Optimizely has a sales team, and I *really* hate that. Nothing like getting a call in the middle of my workday to talk to some vendor.

    So, even if the revelations about Optimizely’s poor setup aren’t enough to dissuade people from buying that product, the high cost ought to be.

  4. We just switched away from using Optimizely and we’re now using another A/B testing product.

    I’m pretty sure that the Optimizely snippet loads asynchronously and has been for some time now.

    As far as I know, the paused variations need to be in the snippet because it’s still shown to visitors which were bucketed into that variation previously, for a consistent user experience, so this is a feature, not a bug or vulnerability.

    I’m pretty sure that the archived experiments are not included in the snippets.

    Loading a static javascript file from the CDN allows the same static file to be loaded quickly all around the world, which I think it’s a fairly reasonable approach.

    Don’t get me wrong, I’m not particularly happy with Optimizely’s implementation (one of the reasons why we switched), but I think you should double check your facts (especially regarding the synchronous loading).

  5. @Tim – I am in the same situation as Ian. Can you please elaborate on why the vulnerability still exists when the optimizely code isn’t directly visible on the page’s source code?
    Thank you kindly

  6. Ian,

    Unfortunately this is true for anyone who is using Optimizely. It took me less than 10 seconds to figure out that you planning to conduct a campaign named “BOATER: USA Call to Action Regulation Page (1)” on your website and you have 8 campaigns planned for future.

    Yes, everything in the post is relevant to you as well even if you use a Tag Management tool.

  7. sparshuga,

    Thanks. I’m limited in my technical abilities. Can you tell me how you did that? The site turned up nothing when I put in my URL.

  8. @Christina, no problem.
    While you don’t have any Optimizely code visible on your page, it is the job of your tag management system to load the Optimizely code. The vulnerability is that once the Optimizely code runs (whether directly in the page source or through a script loader or tag management system), it loads all of your tests by communicating with the Optimizely servers, thus exposing your tests etc.

  9. @Christina – Even if the code is not directly visible on the page source, it gets loaded up by a tag manager / external script and gets executed (thats how it works). One can easily view those scripts and figure out all past and future campaigns.

  10. @Andy Bernard,

    I’ve highlighted and dragged the snippet to my Chrome bookmark bar, but what do I do with it? Do I navigate to a page and then click it? Nothing happens if I do that. What should occur? Sorry for being so technically illiterate, I’m still learning.

  11. This has been around since forever with Optimizely, and I don’t think it’s that big a deal. So what if you can see what other sites are testing? I think it’s pretty crappy of you guys to try and capitalize on this negative publicity which is primarily fear-driven. Expect Optimizely to do the same to you when something negative inevitably happens with your company. I’ve also found when I’ve seen this sort of “unsportsmanlike conduct” in the past that the company that does it ends up not doing all that well. While I would have and did recently consider VWO as an alternative to Optimizely in the past, I won’t in the future because of this sort of behavior.

    I think a better approach is to continue to expand the overall size of the market since optimization in general is still not standard practice, and publicizing something like this probably hurts you as well by increasing fear among those who oppose testing inside companies. There is simply a lot more to be gained by trying to expand the size of the overall market rather than spending your time spreading FUD.

  12. @Ian

    You can open your website in your web browser and copy paste the below line in your address bar and press enter:


    This will open up a window and give all the details about your optimizely account.

    P.S. The JS code is valid / same for every website using Optimizely.

  13. I agree with your points here, but your post seems to imply that using an external CDN can’t be a good solution to implement a fast solution like you guys have and I don’t think that’s the case. For example, if you used a modern CDN like Fastly (and many others), you could cache all your test configurations by page on their CDN and use their API to instantaneously purge the edge cache any time a customer updated their tests. I’m sure Optimizely could do the same with Akamai if they changed their implementation as well.

  14. A little sad this has come out and everyone knows about this now. We’ve been using this to spy on our client’s competitors for a while, and the “edge” we have is likely going to dull a bit and the best competitive insight tool available (unintentional) might go away (if they fix this).

  15. @Jon

    CDNs are in fact good and much faster way of delivering static content but when it comes to serving dynamic content for every URL, they probably might not be the best fit. We have experimented a lot with different CDNs and figured out that for billions of potential URLs (and they keep increasing every second), its near to impossible to have configuration for all of them at the CDN level.

    Imagine, you want to run a campaign on every page of your e-commerce website and you just added a new product (and hence a new product page), your CDN is not aware of this page and has no configuration for it. Chances are that your AB testing tool is also not aware of this page.

    With an architecture like ours, we can match URLs with rules and generate intelligent content that should be served per request level.

  16. @sparshgupta thanks for the reply.

    I’m curious with your architecture, do you have web + replicated database servers at every location to speed things up or do you use a more typical CDN-like model where your origin might be in one place and only the 2nd request for the similar resource could take advantage of local caching? Or maybe it’s nothing like that 🙂 Either way I’d love to learn more about it – for science!

    From where we are in San Francisco, there’s not much you could do to be any faster but I can’t help but brainstorm how an external CDN might work with VWO also. It would definitely be more complex to have to define rules with another CDN partner but if you cached pages like j.php?a=####&u=…&r=0.8427… and told the CDN to exclude the random part when looking up the cache key and to use GeoIP information to vary the lookup also, it seems like you could get a CDN to cache per page. Then you could tag pages by account and perhaps by other things to facilitate cache clearing when tests are updated since any test’s goals or other parameters could invalidate cached content for any page.

    The CDN wouldn’t need any special configuration per page, it would just know to hold onto the cached copy until you told it to purge it (when the user changed their config). On your end, your app would continue to serve things exactly the same way for the most part, and the CDN would transparently cache stuff closer to users. Even though your current solution seems to rock, I think the biggest advantage of an external CDN would be to accelerate VWO for users in regions where you don’t yet have the scale to add servers of your own.

  17. I’m a long time VWO customer, but this article is HUGELY misleading!

    1. You can see what others are testing with Optimizely, and now what? No one cares… You can do the same by loading the page multiple times in private browsing mode with VWO and get the same result!

    2. Your tracking code is NOT faster. In asynchronous mode, you are hiding content from a website while your tracking code is loading:
    opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important;

    This causes a website not loading progressively on the first load and blinking when changing pages! So your asynchronous code is basically useless. It only protects from a site not loading in case your server crashes. That is not a case with Akamai CDN that Otimizely use. And you know that.

    3. So we come to that the best is to use synchronous code for the A/B testing tool. But your synchronous code is slower. And here is why:
    a) Your code is not cacheable.
    b) You are loading 2 scripts instead of 1.
    c) While your first script, size of 1 KB (compressed), takes around 53ms to load – second script size of 33.5 KB (in my case), takes 303ms to load. Why so slow? Because you are using Amazon CloudFront CDN which is much slower than Akamai and have a very slow DNS (relatively). Total load time – 356ms. Total size 34.5 KB.

    Now lets look at Optimizely: one “large” script of 46.8 KB (compressed) that loads in 37ms. That’s because Akamai is the largest CDN in the world. Total load time – 37ms. Almost 10x faster than yours! And it’s cacheable! Loads only once per site! Optimizely have chosen very wise strategy with more benefits for end users and website performance. I’m a long, long time advocate for this scheme. But you prefer to save money using your own servers and slow Amazon CDN.

  18. As someone previously mentioned – you can technically get this information if you clear cookies and come back multiple times. Might take a while but thats an inherent issue of a/b testing of anonymous users?

    I’d be more worried if you could see the results of the tests!

  19. @Gleb:

    > 1. You can see what others are testing with Optimizely, and now what? No one cares… You can do the same by loading the page multiple times in private browsing mode with VWO and get the same result!

    The issue is knowing what tests you’re planning or you had done in the past, not tests you’re currently running. This might not be a big issue for some, but for many giving away business strategy is a valid concern (imagine if competition knows weeks in advance that you’re going to A/B test increased pricing)

    > 2. Your tracking code is NOT faster.

    The asynchronous method of parallel loading is by definition faster. Browser does not wait for test settings to be downloaded before it downloads rest of the page. Modern browsers open multiple threads for downloading content. So this non-blocking way of downloading is definitely a huge advantage.

    > 2. It only protects from a site not loading in case your server crashes.

    Yes, and this is a big advantage. Check the examples provided above on what happens in the case CDN is slow. The entire website does not load.

    > 3. But your synchronous code is slower.

    You can benchmark our code on watchmouse / site24x7. It’s true it is not cached, but it is fast. If you’re overly concerned about speed, we also have self hosting option where you can upload the files on your servers. Evidently, the process of uploading file son your servers involves extra hassles, but then you save on any external dependence and since you’re serving from your website, it’s super fast (no DNS lookups).

  20. AnkitJain@Wingify Thanks for the video, very helpful and I see your point.

    This shows that the snippet is blocking. If the Optimizely code passes control back to the browser right after execution, I would still consider it as semi-asynchronous.

    This is a mere technicality I guess.

    But yes, having this form of a blocking code is a very bad idea. Don’t know why it’s architected in this way. I’m not a VWO user, but I’m glad companies like yours are paying attention to these issues. Thanks!

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login

Resources Home