Cloudflare outage, 2 November 2023

Cloudflare control plane and analytics outage (Oregon power failure) (2 November 2023). The disruption lasted about 40 hours 41 minutes, according to Cloudflare’s own timeline.

What happened. Flexential’s PDX-04 data center, which hosted Cloudflare’s largest analytics cluster and a third of its high-availability cluster, lost power after a utility feed issue and a ground fault on a PGE transformer, and generators failed. Some Cloudflare control-plane services had non-obvious dependencies on that facility, so they went down even though the HA design should have survived.

What changes. Cloudflare declared ‘Code Orange’: removing control-plane dependencies on core data centers, requiring GA products to use the HA cluster with tested disaster recovery, and running more rigorous chaos testing.

Questions

What caused the Cloudflare outage in November 2023?

Flexential’s PDX-04 data center, which hosted Cloudflare’s largest analytics cluster and a third of its high-availability cluster, lost power after a utility feed issue and a ground fault on a PGE transformer, and generators failed.

Which services were affected?

Cloudflare dashboard, APIs (control plane), logging and analytics; network traffic unaffected.

Read how shared dependencies spread a failure in why one cloud outage can break unrelated apps, or browse the outage archive.