Cloudflare down latest: ‘Fix’ update issued after X, ChatGPT and more websites suffer mass outage

Alan Woodward, professor of cybersecurity at the University of Surrey said: “The downside of being a gatekeeper and distribution network for such big brands is that if this vital system fails, no one can use your service be that website or app.

“It’s still not clear exactly what went wrong but it looks like it was a technical malfunction within Cloudflare network.

“This in itself is surprising as such networks are designed to avoid single points of failure.”

Cloudflare provides network and security services for many online businesses in order to help their websites and applications operate.

This includes running services such as checking that visitors to websites are humans rather than bots, says around a fifth of all global websites use some of its services.

Cloudflare had scheduled maintenance for the SCL (Santiago) data centre for Tuesday.

While Cloudflare has succeeded in returning to its fully operational status by this evening, it’s large scale outage earlier today has prompted questions how likely it is that such an incident will occur again.

Lee Skillen, CTO of software artefact management platform Cloudsmith, said: “Although outages are not uncommon, a global "completely down and out" outage like this is absolutely, highly unusual, and there is no doubt that this has a wide-reaching impact worldwide for businesses and their users.

“Modern infrastructure is built on deeply interconnected systems; the more we optimise for scale, the more challenging it becomes to pinpoint how one failure cascades into another. Will this happen more frequently? The shower answer is yes. Expect things to fail.

“They can range from mild and short-lived to rare and catastrophic. But they all share one thing: inevitability. Every service with real users will eventually get hit by something; sometimes directly, sometimes indirectly. The greater the magnitude, the greater the possibility.

“Today, it's Cloudflare, and tomorrow it may be Fasty, or one of the Cloud provider CDNs. Each is a reminder of how architectural choices ripple outward, exposing an implicit "supply" chain. This is why critical infrastructure providers know they’ve got a (difficult) but important job to do, and invest millions in ensuring that it doesn’t happen.”

In an updated notification to their system, the network service said: “Cloudflare services are currently operating normally. We are no longer observing elevated errors or latency across the network.

“Our engineering teams continue to closely monitor the platform and perform a deeper investigation into the earlier disruption, but no configuration changes are being made at this time.

“At this point, it is considered safe to re-enable any Cloudflare services that were temporarily disabled during the incident. We will provide a final update once our investigation is complete.”

Dane Knecht, Cloudflare’s chief technology officer, has said that their services should now be “fully operational” in his latest update after the internet infrastructure suffered a technical problem.

He said: “Sharing an update on the recovery of our services. We were able to resolve the impact to traffic flowing through our network at approximately 14:30 UTC, which was our first priority, but the incident required some additional work to fully restore our control plane (our dashboard and the APIs our customers use to configure Cloudflare).

“The control plane should now be fully available. We are monitoring those services and continuing to ensure that everything is fully operational. Again, we plan to share a complete walkthrough of what went wrong today in a couple of hours and how we plan to make sure this never happens again.”

Today felt very dramatic. But it’s nowhere near the biggest outage of recent years. Here’s a rundown of some of the worst.

Dane Knecht, Cloudflare’s chief technology officer, is direct and damning in his words about the outage today. In a long post on X, formerly known as Twitter, he admitted that his company had “failed” its customers.

“I won’t mince words: earlier today we failed our customers and the broader Internet when a problem in @Cloudflare network impacted large amounts of traffic that rely on us. The sites, businesses, and organizations that rely on Cloudflare depend on us being available and I apologize for the impact that we caused,” he wrote.

“Transparency about what happened matters, and we plan to share a breakdown with more details in a few hours. In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack.

“That issue, impact it caused, and time to resolution is unacceptable. Work is already underway to make sure it does not happen again, but I know it caused real pain today. The trust our customers place in us is what we value the most and we are going to do what it takes to earn that back.”

The latest update from Cloudflare suggests that there might still be some issues as the company and its customers recover.

“The team is continuing to focus on restoring service post-fix. We are mitigating several issues that remain post-deployment.”

This comment from Alan Woodward, Professor of Cybersecurity at the University of Surrey, explains both what Cloudflare is and why today’s outage spread so widely.

“Cloudflare provides a form of internet shield, preventing some forms of attack and unwanted visits from bots, as well as acting as a global distribution network for content for its clients. Those clients include some well known names such as X, Spotify and Zoom. It supports something like 30% of the Fortune 100 companies. The downside of being a gatekeeper and distribution network for such big brands is that if this vital system fails, no one can use your service be that website or app. It’s still not clear exactly what went wrong but it looks like it was a technical malfunction within Cloudflare network. This in itself is surprising as such networks are designed to avoid single points of failure.

“Although Cloudflare say their services are recovering, the high profile clients means it was highly visible and disruptive. Knock-on effects are still being felt as Cloudflare continue to investigate what happened.

“This incident, as with the recent outage at AWS, shows how reliant some very important Internet based services are on a relatively few major players. It’s a double edged sword as these service providers need to be large to provide the scale and global reach required by big brands. But when they fail the impact can be significant. This won’t be the last time one of these big providers fails and the internet suffers disruption.”

Anecdotally, it does seem that Cloudflare’s update (see below) is correct. Websites that had previously been broken such as Twitter and ChatGPT are now back to life.