Incidents history
October 2023 to August 2023
October 2023
Cloudflare: Cloudflare Pages, Workers KV and Cloudflare Access Availability Issues
We have further analyzed what happened in this incident and here's a longer explanation.
We use Cloudflare workers at the edge of our platform to cache GraphQL requests and block traffic that goes over hard limits, for example for people with cancelled subscriptions. To block these projects we save the information in the Cloudflare KV.
When Cloudflare KV was down our workers were breaking causing the cache to be bypassed.
So all the GraphQL requests were going straight to our servers causing an unusual load that slowed down everything very significantly. Only a minority of requests timed out, but still the user experience was significantly affected and everything seemed down. Furthermore, because everything hit our servers the rate-limits for GraphQL requests were hit much more frequently.
To avoid this from happening in the future we are going to improve our workers to avoid them for breaking in case of the KV service being down. If this has to happen again at least the caching will keep working. We didn't anticipate that the workers and KV storage might break independently, but we'll improve this part of the infrastructure soon.
September 2023
Unresponsive Real-Time APIs
This incident is now resolved. The impact start time was 2023-09-27 18:30 UTC, and the end time was 2023-09-27 20:10 UTC.
Rollbar security notice
On September 8th Rollbar announced a security incident.
Rollbar is our error monitoring tool, where we send stacktraces of errors collected on frontend and backend systems.
All the details of the incident here: https://rollbar.com/compliance/nob_sep7th/
In brief, our Rollbar access tokens might have been accessed by a third party, but Rollbar has no evidence of this happening. This means that our traces could have been read by someone else. In our implementation, before sending the traces to Rollbar we strip any personal information that might be present or any access token to DatoCMS projects.
Apart from the posted data in the traces nothing else could have been exposed to third parties.
In any case, on Sep 12th we have rotated all our tokens and deleted all the stack traces that were present in Rollbar. To this day we don't have any evidence that any of our data was accessed by a malicious third party.
August 2023
No incidents reported.