The Traffic Spike You Scheduled: Auditing Synthetic Checks Against Origin Rate Limits
Photo: Mike Reyher, CC BY 2.0, via Wikimedia Commons
An HTTP 429 is often described as a capacity signal. In synthetic monitoring, it can be a scheduling artifact. A monitor that probes every minute from several US regions, using a shared egress range and no jitter, can exceed a rate limit that no human visitor would ever touch. The check then reports an outage that exists only for the monitor. The useful question is not whether a 429 occurred, but whether the monitor caused it, and whether the origin is also rejecting real traffic.
How a monitor becomes the traffic spike
Rate limits are usually keyed by IP, ASN, account, cookie, or endpoint. Synthetic checks often share a small pool of egress addresses, omit cookies, and carry a consistent user agent. Multiply that by probe frequency, region count, and endpoint count, and the monitor can look like a small botnet. Minute-boundary schedules make it worse: five regions fire at 10:00:00, retry at 10:00:02, and collide again. A WAF rule or CDN rate limiter does not know the request is a health check unless it has been told.
Read Retry-After literally and completely
The Retry-After header can be a delta-seconds value or an HTTP-date. Parse both forms, store the raw header, and log whether it was absent, zero, or implausibly large. A missing header does not mean retry immediately. A value of zero is not permission to hammer the endpoint. If the response also includes rate-limit headers such as RateLimit-Reset or X-RateLimit-Remaining, record them alongside the status code. The monitor should honor Retry-After with jitter, and it should stop retrying a single endpoint after a bounded number of attempts. Otherwise the retry policy becomes a second load test.
Separate self-inflicted 429s from real pressure
Look for evidence that the 429 is monitor-specific. Does the status appear only for the monitor user agent or egress range? Do origin logs show the probe cadence rather than user sessions? Does a controlled reduction in one region's frequency make the 429 disappear? Are real-user monitoring or CDN logs showing 429s at the same time? If real users are also rate-limited, the problem is capacity or policy, not the monitor. If only synthetic checks are affected, the monitor is likely exceeding a limit that was never sized for it.
Do not simply allowlist the monitor and close the ticket. An allowlist can hide a policy that also affects legitimate high-volume clients, and it can mask a genuine origin limit during peak traffic. If an allowlist is necessary, document it, scope it narrowly, and keep a second canary probe outside the allowlist so the monitor still sees the public behavior.
Stagger probes without blinding the monitor
Distribute probes across regions with per-region jitter and offset schedules. Cap concurrency per origin and per endpoint. Use exponential backoff with jitter after 429s, and use a circuit breaker per region so one rate-limited region does not create a retry storm. Keep at least one low-frequency canary on a stable schedule so a global slowdown is still visible. If the monitor supports it, rotate probe order so the same endpoint is not always first.
Staggering should not become a way to avoid alerts. Keep a baseline frequency that is high enough to detect real failures, and alert on the rate of 429s after backoff rather than on a single event. A single 429 during a deploy may be noise; a sustained pattern across regions is a finding.
Build a rate-limit budget
Document the origin's published limits, the monitor's request budget, and the headroom left for real users. Include CDN, WAF, and application-layer limits, because a request can be rejected at any layer. Review the budget when probe frequency, region count, or endpoint count changes. During peak 2026 traffic, reduce nonessential probe frequency before reducing coverage of checkout and login paths.
Alert on the signal that matters
A monitor-induced 429 is a false outage. A real-user 429 is a genuine degradation. The dashboard should distinguish them. Alert on synthetic 429s only when they persist after Retry-After and backoff, when they coincide with real-user 429s, or when latency and 5xx rates rise at the same time. Keep the raw evidence: status, Retry-After, region, egress identity, endpoint, and timestamp. The goal is not to suppress 429s. It is to make sure the 429 you see is the one your visitors would see.