Cloudflare Falls Over and Takes X, ChatGPT and Spotify With It
A database permissions change, an oversized config file, and six hours of 5xx across a large slice of the web.
On 18 November 2025 a large part of the web stopped answering. Requests to sites and applications sitting behind Cloudflare came back as HTTP 5xx errors instead of pages. The first customer traffic errors appeared at 11:28 UTC and the last services were not restored until 17:06 UTC — roughly five and a half hours. In its own published account, Cloudflare called it the company's worst outage since 2019.
There was no attacker. Cloudflare stated plainly that nothing malicious was involved, neither an attack nor any other deliberate act. What actually happened was smaller and far more ordinary, which is precisely why it is worth reading.
At 11:05 UTC a permissions change went out to a ClickHouse database cluster. Access that users already had implicitly was made explicit, so that the metadata of the underlying tables became visible too. A query that built a configuration file for Cloudflare's Bot Management system had never named which database it wanted. After the change it saw the same columns twice, once from each database, and the file it generated came out at roughly double its normal size.
Then it met a hard limit. The software reading that file preallocates memory for at most 200 machine-learning features, comfortably above the roughly 60 actually in use. The doubled file went past 200, and the code did not degrade gracefully — it panicked, and every request the affected proxy handled returned a 5xx.
Why the cause was so hard to see
The file was regenerated every five minutes, and the database cluster was being updated node by node. A bad file appeared only when the query happened to run on an already-updated node. So for a while the network alternated: five minutes broken, five minutes healthy, five minutes broken again. Sites came back and fell over repeatedly.
That pattern looks nothing like a bad deploy and a great deal like an attack. Cloudflare's status page runs on separate infrastructure with nothing shared with the network it reports on, and it went down at the same time by pure coincidence, which reinforced the theory that someone was hitting them on several fronts at once. The first part of the response was spent on the wrong hypothesis.
Only once every node had been updated, and the failure became constant instead of intermittent, did the real chain become visible. A bypass for Workers KV and Access went in at 13:05 UTC, generation of the bad file was stopped at 14:24 UTC, a correct configuration was deployed worldwide at 14:30 UTC, and the remaining services — the dashboard and Turnstile among them — came back over the following two and a half hours.
What this means if you are building something
Your status page cannot live where your product lives. Cloudflare keeps that page off its own infrastructure, which is the correct decision even though the page happened to fail that day for unrelated reasons. If your site and the page where you announce that your site is down share a provider, you have no way to talk to your customers on the worst day of the year.
Data your own systems generate is still input. Nobody validated that configuration file because it came from inside the house. The first item on Cloudflare's own remediation list is to treat configuration files the company generates itself with the same suspicion it already applies to anything a user submits. The same applies to a price feed, a translation file, or a product export from a client's ERP.
Fail soft, not hard. In the same incident, an older version of Cloudflare's proxy engine hit the same bad file and simply assigned every request a bot score of zero. The newer one crashed. Wrong output that keeps the page up is usually better than a correct refusal to serve anything at all.
Agree the plan before you need it. Kill switches for individual features, cached fallbacks for anything fetched from a third party, and a short written answer to "who says what to customers, on which channel, within how many minutes" are cheap to prepare and impossible to improvise at 11:30 on a Tuesday.
Our reading of the day is not that Cloudflare was careless. It is that a routine permissions change, a query missing one filter, and a limit set more than three times higher than anyone needed were enough to take out a slice of the internet for an afternoon. Cloudflare's post-mortem contains a plain sentence — "We know we let you down today." The useful response is not to pick a different provider. It is to decide, in advance, how your project behaves when the provider you chose has a bad afternoon.
Sources
