Sorry, We Messed Up
It has come to our notice that a new beta feature that we switched on for PushCrew customers violated one of the principles we hold very dearly: our customer data privacy. By putting PushCrew code (now VWO Smartcode) on your website, you place a trust in us, which we do not take lightly, and that’s why it’s important for you to know about an incident that we came to know in last few hours. The short version is that because of a mistake on our part, user data from your website was being sent to our servers since 20th April 2017, and when a customer made us aware of this, we realized our mistake, tracked down the erroneous piece of code and removed it. We then immediately, permanently and irreversibly deleted all copies of the data on our servers and machines (including the automatic backups).
This update is about what happened, who was impacted, and what we have done about it.
One of the upcoming PushCrew features enables sending notifications based on visitor behaviour on customers’ websites. For example, sending notifications when a visitor abandons a cart, visits the best seller page, etc. To enable this feature, we use an open source library called SnowPlow to collect events on a website so that automatic notifications can be sent. Snowplow has a setting which enables collection of form data, which could contain sensitive information such as email, phone numbers or credit card information. By default, this setting is off but during prototyping stage, our engineers tried various settings of the library and due to slippage on our part, form tracking setting remained enabled (which we should have disabled). While pushing the automatic notifications feature out to be ready ahead of beta testing, we wanted to load test the new library on production systems and enabled the SnowPlow derived library for 400 users. The flag for collection of form data remained enabled in production environment, and a customer alerted us that sensitive data was being sent to our servers.
This is a major mess-up and we apologise. We would never want to misuse the trust that customers have put in us and we certainly did not intend to collect sensitive data without customer’s explicit permission.
Who got impacted
We activated this library on 20 April 2017 date and for 400 customers. Since then, this library was collecting data and sending it to our servers. We did not realise that along with the data we wanted (say which page the visitor was visiting), it might also have been sending sensitive data such as credit card numbers or passwords.
What we did
– We immediately deactivated (and are in the process of removing the library completely from the VWO Smartcode).
UPDATE: As of noon on 6th May IST, we have removed the library completely from the VWO Smartcode.
– We deleted all the data that was collected by this library (sensitive or non-sensitive) from our servers and have discarded the entire machine where this data was being collected.
– We deleted our access logs that could have had traces of data.
– We have contacted our backup provider and have asked them to remove the data from backups.
– We contacted the affected customers and users informing about the incident.
– The data transfer between customer websites and our server happened on secure HTTPS protocol, which means that it’s highly unlikely that someone else could have gotten access to the data, but we’re still thoroughly investigating that possibility.
– We have changed the relevant encryption keys and deleted the old ones. This ensures that even if the data is retained, we are unable to decrypt it.
Reiterating, so that this is clear: data WAS NOT leaked to any external party and your customers’ data is safe with you. Sensitive data was simply getting recorded on our servers without us realising because of a setting in a code library we use that should not have gotten to our production environment. After full deletion of the encrypted data (including backups), we can confirm that nobody in our organization has access to your or your customers’ data. Moreover, we are confident that no 3rd party could get access to that data because: a) communication over HTTPS is secure to man-in-middle attacks; b) the user data on our servers was encrypted (so even if in the most unlikely situation anyone got access to that data, they won’t be able to decrypt it without keys that we have, and we immediately deleted the keys to prevent that from happening).
What we’re doing to prevent such incidents in future
We’re dedicating extra bandwidth and resources to hiring a dedicated customer data privacy officer and are also building internal checklists and audit processes for such situations. We want to nullify the possibility of this ever happening again.
What’s PushCrew’s stance on user and customer data?
Who to contact for more details
You can contact our support at email@example.com or if you want to contact me directly (the CEO), you can email me at firstname.lastname@example.org
We will keep updating this post as and when we have more information.