Sorry, We Messed Up

It has come to our notice that a new beta feature that we switched on for VWO Engage (formerly PushCrew) customers violated one of the principles we hold very dearly: our customer data privacy. By putting VWO Engage code (now VWO Smartcode) on your website, you place a trust in us, which we do not take lightly, and that’s why it’s important for you to know about an incident that we came to know in last few hours. The short version is that because of a mistake on our part, visitor data from your website was being sent to our servers since 20th April 2017, and when a customer made us aware of this, we realized our mistake, tracked down the erroneous piece of code and removed it. We then immediately, permanently and irreversibly deleted all copies of the data on our servers and machines (including the automatic backups).

Download Free: A/B Testing Guide

This update is about what happened, who was impacted, and what we have done about it.

What happened

One of the upcoming VWO Engage features enables sending notifications based on visitor behaviour on customers’ websites. For example, sending notifications when a visitor abandons a cart, visits the best seller page, etc. To enable this feature, we use an open source library called SnowPlow to collect events on a website so that automatic notifications can be sent. Snowplow has a setting which enables collection of form data, which could contain sensitive information such as email, phone numbers or credit card information. By default, this setting is off but during prototyping stage, our engineers tried various settings of the library and due to slippage on our part, form tracking setting remained enabled (which we should have disabled). While pushing the automatic notifications feature out to be ready ahead of beta testing, we wanted to load test the new library on production systems and enabled the SnowPlow derived library for 400 users. The flag for collection of form data remained enabled in production environment, and a customer alerted us that sensitive data was being sent to our servers.

This is a major mess-up and we apologise. We would never want to misuse the trust that customers have put in us and we certainly did not intend to collect sensitive data without customer’s explicit permission.

Who got impacted

We activated this library on 20 April 2017 date and for 400 customers. Since then, this library was collecting data and sending it to our servers. We did not realise that along with the data we wanted (say which page the visitor was visiting), it might also have been sending sensitive data such as credit card numbers or passwords.

What we did

– We immediately deactivated (and are in the process of removing the library completely from the VWO Smartcode).

UPDATE: As of noon on 6th May IST, we have removed the library completely from the VWO Smartcode.

– We deleted all the data that was collected by this library (sensitive or non-sensitive) from our servers and have discarded the entire machine where this data was being collected.

– We deleted our access logs that could have had traces of data.

– We have contacted our backup provider and have asked them to remove the data from backups.

– We contacted the affected customers and users informing about the incident.

– The data transfer between customer websites and our server happened on secure HTTPS protocol, which means that it’s highly unlikely that someone else could have gotten access to the data, but we’re still thoroughly investigating that possibility.

– We have changed the relevant encryption keys and deleted the old ones. This ensures that even if the data is retained, we are unable to decrypt it.

Reiterating, so that this is clear: data WAS NOT leaked to any external party and your customers’ data is safe with you. Sensitive data was simply getting recorded on our servers without us realising because of a setting in a code library we use that should not have gotten to our production environment. After full deletion of the encrypted data (including backups), we can confirm that nobody in our organization has access to your or your customers’ data. Moreover, we are confident that no 3rd party could get access to that data because: a) communication over HTTPS is secure to man-in-middle attacks; b) the user data on our servers was encrypted (so even if in the most unlikely situation anyone got access to that data, they won’t be able to decrypt it without keys that we have, and we immediately deleted the keys to prevent that from happening).

Download Free: A/B Testing Guide

What we’re doing to prevent such incidents in future

We’re dedicating extra bandwidth and resources to hiring a dedicated customer data privacy officer and are also building internal checklists and audit processes for such situations. We want to nullify the possibility of this ever happening again.

What’s VWO Engage’s stance on user and customer data?

Our stance on user and customer data has always been that we will never collect sensitive information for our customers, and will never sell even non-sensitive data to any third party provider. We as consumers value our data and privacy, and we want to hold our customers’ data with the same regard. You can access our official privacy policy here.

Who to contact for more details

You can contact our support at support@vwo.com or if you want to contact me directly (the CEO), you can email me at paras@wingify.com

We will keep updating this post as and when we have more information.