Roblox outage, 28 October 2021
The Roblox outage of 28 October 2021: what failed, why, and what Roblox changed, from its own incident report.
- Provider
- Roblox
- Date
- Duration, hours
- 73
- Cause
- Software bug
- Regions
- Global (Roblox data centers)
Roblox 73-hour outage (Consul/BoltDB) (28 October 2021). The disruption lasted about 73 hours, according to Roblox’s own timeline.
What happened. Enabling a relatively new Consul streaming feature under unusually high read/write load caused heavy contention, and the load also triggered a pathological performance issue in BoltDB inside Consul. One Consul cluster served many workloads and monitoring depended on it, so diagnosis and recovery took 73 hours.
What changes. Roblox is removing circular dependencies in observability, speeding bootstrapping, moving to multiple availability zones/data centers, and fixing the Consul issues.
Questions
What caused the Roblox outage in October 2021?
Enabling a relatively new Consul streaming feature under unusually high read/write load caused heavy contention, and the load also triggered a pathological performance issue in BoltDB inside Consul.
Which services were affected?
Roblox platform (all services).
Read how shared dependencies spread a failure in why one cloud outage can break unrelated apps, or browse the outage archive.
- Services affected
- Roblox platform (all services)
- Incident report
- blog.roblox.com
- Sources
Source Link Roblox incident report https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/