Google Cloud outage, 25 April 2023

Google Cloud europe-west9 (Paris) water leak and fire (25 April 2023). The disruption lasted about 27 hours 14 minutes, according to Google Cloud’s own timeline.

What happened. A cooling-system water pipe leak in a non-Google part of the facility flooded a UPS room and caused a fire, forcing a power shutdown of an entire data center building. Regional Spanner replicas were not correctly spread across the region’s three buildings, so the regional control plane lost quorum, and GCE fanout methods caused global Cloud Console issues.

What changes. Google is auditing Spanner allocations in every region for fault isolation, speeding IAM policy refresh, improving GCE control-plane degradation, and analyzing water-intrusion risk in all data centers.

Questions

What caused the Google Cloud outage in April 2023?

A cooling-system water pipe leak in a non-Google part of the facility flooded a UPS room and caused a fire, forcing a power shutdown of an entire data center building.

Which services were affected?

Multiple Google Cloud services in europe-west9 (GCE, Persistent Disk, Spanner, IAM, GCS, BigQuery, Pub/Sub, Dataflow); global Cloud Console and GCE fanout APIs.

Read how shared dependencies spread a failure in why one cloud outage can break unrelated apps, or browse the outage archive.