AWS outage, 7 December 2021

AWS us-east-1 internal network congestion outage (7 December 2021). The disruption lasted about 6 hours 52 minutes, according to AWS’s own timeline.

What happened. An automated activity to scale capacity of an AWS service triggered unexpected behavior from many clients in AWS’s internal network, causing a surge of connections that overwhelmed networking devices between the internal and main AWS networks. A latent issue stopped clients from backing off, and the congestion also blinded internal monitoring, slowing recovery.

What changes. AWS disabled the triggering scaling activities, committed to fix the client back-off bug, and deployed extra network configuration to protect the affected devices.

Questions

What caused the AWS outage in December 2021?

An automated activity to scale capacity of an AWS service triggered unexpected behavior from many clients in AWS’s internal network, causing a surge of connections that overwhelmed networking devices between the internal and main AWS networks.

Which services were affected?

EC2 APIs, Route 53 APIs, AWS Console, STS, API Gateway, EventBridge, CloudWatch, VPC Endpoints for S3/DynamoDB, RDS/EMR/WorkSpaces provisioning.

Read how shared dependencies spread a failure in why one cloud outage can break unrelated apps, or browse the outage archive.