Contellect | Empower your business

We recently discovered a data ingestion issue on the Mixpanel platform that affected a small subset of certain customers’ projects. Affected customers were previously notified via direct email, and they should update their websites to use the new Mixpanel JavaScript SDK as soon as possible (unless their SDK is set to automatically update, or they have done so already).

We want to explain what happened, how we’ve worked to fix it and what we’re doing to ensure that something like this doesn’t happen again.

Before we go into these details, we want to apologize to all of our customers. None of this should have happened, and we will not make excuses for it. Our priority is to ensure that our customers’ data is secure at all times, and we are conducting a thorough investigation of what happened and how we handled this.

What happened?

On January 5th, 2018, a customer informed us that they observed Autotrack sending the values of password fields in events. We confirmed that this was unexpected behavior; by design, Autotrack should not send the values of hidden and password form fields.

We immediately began investigating further and learned that the behavior the customer was observing was due to a change to the React JavaScript library made in March 2017. This change placed copies of the values of hidden and password fields into the input elements’ attributes, which Autotrack then inadvertently received. Upon investigating further, we realized that, because of the way we had implemented Autotrack when it launched in August 2016, this could happen in other scenarios where browser plugins (such as the 1Password password manager) and website frameworks place sensitive data into form element attributes.

As stated above, our forensics and security experts have not seen any indication that this data was downloaded or accessed by any Mixpanel employee or third party. Upon discovery, we took immediate steps to secure the data and shut down further receipt. As of February 1st, all data that was inadvertently received has been destroyed. In order to be as transparent as possible, here is more detail on how we have addressed and will continue to address this issue.

How we’re addressing this issue

Since discovery, we have been actively working to resolve the issue for affected customers. Only about 4% of Mixpanel projects were impacted, and we have reached out directly to those customers with impacted projects.

We took immediate steps when we discovered this data ingestion issue in order to:

1. Limit further receipt of data: On January 9th, we implemented a server-side filter to securely discard this data as soon as we receive it, and soon thereafter refined the filter to solve for the last remaining edge cases.
2. Delete the inadvertently received data: We have cleared all data from our database that we inadvertently received and, upon request, have offered affected customers fine-grained metadata about what data was inadvertently sent to Mixpanel servers.
3. Fix the Autotrack bug: We have implemented the Autotrack functionality fix in the Mixpanel JavaScript SDK. Affected customers will, however, need to update their Mixpanel JavaScript SDK as soon as possible to reflect this change (although 85% of affected customers already have updated). If your SDK is set to automatically update, or if your website loads the SDK directly from our content servers, then no action is required.
4. Review any access of this data: We do not believe this data was downloaded or accessed by any Mixpanel employee or third party. To the extent we discover otherwise, we will immediately notify our customers.

In addition to fixing the root cause of this issue, we’re taking proactive steps to identify and prevent similar issues from occurring in the future.

1. Incorporating additional privacy reviews as part of our design and development processes: Security and privacy have always been front of mind at Mixpanel, but we’re adding some additional explicit checkpoints in our product development processes to help ensure that we’ve considered all of the impacts of the changes we make.
2. In-depth security/privacy audits of key existing product areas: We’ve learned a lot from this issue, and our team has been diving in to look for similar cases where these same kinds of problems could arise.
3. Operationalizing our response tooling: We’ve built new tools in response to this issue to help us identify the scope of data collection, limit access to data, and to purge it from our systems quickly. We’re taking these tools and making them more general purpose so that we can respond more quickly in the unlikely event that a similar problem occurs in the future.
4. Data filtering and detection: We’re exploring capabilities that can detect something like this sooner including changes to the SDK to give us more insight into what data is being sent to us, integration with Data Loss Prevention (DLP) solutions, and even using our machine learning capabilities to detect anomalous ingestion.

Moving forward

We are continuing to conduct a thorough investigation of what happened and how we handled it. We believe that we have addressed the ingestion issue with the speed and accuracy required as your trusted partner.

As part of our action plan, we’ve disabled Autotrack by default for all new projects created, and we’ve put the product on hiatus for now. Ultimately, all of our products and features must put the privacy and security of customers’ data first. If we conclude that certain features cannot meet these high standards, even if they provide value to our customers, we will modify or remove them entirely.

If you have questions or for more information, please email “autotrackquestions@mixpanel.com”, and we’ll be in touch as soon as possible.