Increased number of timeouts

Incident report for DatoCMS

Resolved

We have identified and improved the query that has caused a snowball error in our database bringing most of the queries to timeouts.

The total number of record versions saved in our DB is growing fast and it's causing new challenges for us. We'll monitor the situation in the next few days. Hopefully we'll be faster next time as we have also improved our error reporting from the DB.

Posted at Sep 24, 15:59 GMT+00:00

Identified

We have found the root cause in a slow query that was bringing down the database for everyone.

We have limited the query duration and the status seems back to normal.

We are monitoring and trying to release a stable fix.

Posted at Sep 24, 15:20 GMT+00:00

Investigating

We are seeing an increased number of calls timing out.

When we see a spike of timeouts we can see errors across different endpoints.

We are investigating the root cause of the issue and we'll report back as soon as possible.

Posted at Sep 24, 14:31 GMT+00:00