All affected gateway health checks have recovered. The incident is resolved. Please see the post-incident review in this report for the cause and corrective actions.

Resolved — Post-incident review

Between 00:54 and 01:04 UTC on August 1, health checks for our Europe and United States proxy gateways returned HTTP 503 responses. Gateway availability was degraded during this period, and some proxy requests may have failed or experienced increased latency.

Cause

During an internal analytics investigation, an unbounded high-cardinality query was run against the production analytics datastore on a shared host. The query consumed nearly all available memory while also saturating CPU and storage I/O. This prevented shared health dependencies from responding normally and caused the analytics service to restart.

Resolution

The analytics service restarted, the resource pressure cleared, and all gateway health checks recovered automatically by 01:04 UTC. We found no evidence of data loss or a security impact.

Corrective actions

• Add strict per-query memory, runtime, and concurrency limits for analytics and diagnostic workloads.
• Reduce the analytics datastore's host-wide memory allowance so the operating system and proxy services retain sufficient headroom.
• Move RPS analysis and alerting to pre-aggregated data instead of raw multi-day scans.
• Add alerts for host memory pressure, CPU saturation, disk I/O pressure, and datastore restarts.
• Review workload isolation for the proxy API and analytics datastore.

We apologize for the disruption.

We are investigating HTTP 503 responses affecting health checks for multiple Europe and United States proxy gateways. Gateway availability may be degraded.