Previous incidents

The incident has been resolved.

We are back!

The issue was resolved.

The system has recovered. We have escalated the root cause to our upstream provider.

We are back.

Upstream provider has resolved the issue.

firecrawl.dev recovered.

We are back. Job timeout metrics have recovered to pre-incident levels.

The issue came down to misconfigured pipeline queue limits on Dragonfly -- we configured them with high values expecting a high load on production, however, they ended up being ridiculously high. This caused Dragonfly's backpressure mechanisms to kick in way too late, only when the state of the instance is practically already unsalvageable. The configuration was tuned, and we will continue to monitor this.

The API has recovered.

We quickly restarted the workers to get them un-stuck again. After that, we revisited the core issue. A lot of jobs were finishing at the same time, started hammering the same sorted set in Redis at the same time, and ran into the same race conditions. We decided to break up these clumps of requests by applying a timeout of random length before retrying every time the anti-race mechanism activates. This spreads out the workers nicely.

After applying the fix and observ...

Service is restored. Crawls that appeared "stuck" should now resume.

The issue is fully resolved. Apologies for the disruption and thank you for your patience.

The issue was caused by a load spike, triggering a scale-up to a high amount of API and Worker pods. These pods interface heavily with our Dragonfly (Redis-equivalent) instance via BullMQ for job queueing. The increased connections and requests to Dragonfly caused it to start having to queue pipeline operations, which caused BullMQ operations to have a delay, making the system fail and scrape jobs accu...

We're fully back up now.

Some crawls started between 9:30 PM EST and 10:30 PM EST may have been affected by the outage. If you need help, please reach out to help@firecrawl.com.

We sincerely apologize for the disruption in some requests. The issue was caused by a sudden surge in traffic (100x normal amount) that overwhelmed our internals and upstream providers. We're continuing to work with them to ensure this doesn't happen again.

A full incident report will be shared later this week.

The issue is resolved. We are investigating further.

We are fully back. We have also managed to upgrade our infrastructure during this time to avoid the errors experienced today in the future. Thank you for your patience.

The issue has been resolved for now and all services are fully operational. We're currently working with the GCP team as they continue investigating the root cause. Once their analysis is complete, we’ll publish a detailed incident report outlining what happened and the steps taken.
Thank you!

We are back!

All /extract calls are working again.

The API has recovered.