# How should we alert on a critical endpoint that gets only a few requests per hour? > For a critical endpoint with only a few requests per hour, do not page on a short-window error percentage alone. Canonical URL: https://www.devobs.io/articles/qa-low-traffic-alert-confidence/ By: Lucas Vale Published: 2024-04-12T18:32:13.054Z Updated: 2026-09-06T08:31:04.426Z Section: Architecture Do not page on a short-window error-rate percentage alone when an endpoint gets only a few requests per hour. With such a tiny denominator, one failure can mean 50% or 100% error rate without proving sustained user harm. A better pattern is to combine three signals: immediate evidence of a hard failure mode, synthetic coverage for the workflow you cannot afford to miss, and a longer statistical window for trend confirmation. That matches the SRE tradeoff between precision, recall, detection time, and reset time described in the [Google SRE Workbook on alerting](https://sre.google/workbook/alerting-on-slos/). ## What can a tiny request count actually tell you? A sparse endpoint can tell you that a single request failed. It usually cannot tell you, from percentage alone, whether the service is in a broad incident. The [Google SRE Workbook on alerting](https://sre.google/workbook/alerting-on-slos/) explicitly warns that alerting becomes sensitive to nonsignificant events during low-traffic periods and evaluates strategies by precision and recall. That is the key boundary here: collect the evidence first, then page only on the evidence that actually proves urgent harm. A 1-of-1 or 1-of-2 failure rate is real data, but it is weak aggregate evidence. So separate alerts into two classes: - **Page now:** a definitive bad event on a critical path, such as every synthetic attempt failing, repeated 5xx responses with no recent successes, or an endpoint that must always succeed for a business-critical operation. - **Ticket or investigate in hours:** elevated failure percentage across a longer window, where you are measuring reliability trend rather than acute outage. ## Which signals should you combine? First, use **direct failure evidence**. Alert on a hard symptom such as consecutive failed requests, specific fatal status codes, or a queue of user-visible failed jobs if that endpoint drives asynchronous work. This avoids pretending that a percentage from two requests is statistically strong. Second, add **synthetic checks** for the exact workflow. If the endpoint is rarely called but high value, a controlled probe gives you the missing denominator. This is the fastest way to improve recall without waiting for real users. Third, keep a **longer window SLO-style alert** for confirmation and trend. The [Google SRE Workbook on monitoring](https://sre.google/workbook/monitoring/) recommends monotonically increasing counters and notes that computing rates over a longer window provides the building blocks for burn-based alerting. For sparse traffic, that means hours or a day, not five minutes. ## How should this look in practice? Use a simple decision checklist: 1. **Ask how long detection can wait.** If the answer is “minutes,” add a synthetic probe and page on consecutive failures. 2. **Define a hard failure signal.** Example: page if 3 consecutive synthetic attempts fail or if 2 real requests in a row return 5xx with no success in between. 3. **Add a long-window reliability signal.** Example: create a warning when 24-hour success rate drops below target or when error-budget burn stays elevated over several hours. 4. **Tune reset behavior.** Require one or two successful synthetic runs before auto-resolving to avoid flap. Worked example: suppose `/funds/transfer/confirm` receives about 4 requests per hour. Do **not** page on “error rate over 5 minutes > 1%.” One failed request would page too often for poor reason. Instead, run a synthetic transfer-confirmation every 2 minutes against a safe test account, page after 3 consecutive failures, and open a daytime investigation if the 24-hour success rate degrades. That gives you immediate outage detection plus a statistically meaningful trend view. ## What if synthetics cannot perform the action? Use the closest safe precondition and instrument downstream evidence. For example, probe authentication, validation, and dependency reachability separately, then combine that with failed real-event counters. If no safe synthetic exists, accept slower detection and rely more on long windows and strong hard-failure symptoms. ## How long can detection wait? Set the window from business impact, not from a generic percentage threshold. If a missed request is intolerable, you need immediate synthetic evidence. If the endpoint matters but occasional retries are acceptable, a multi-hour or daily statistical window is more honest. Next step: write down one paging condition based on hard failure evidence and one non-paging long-window reliability condition for the endpoint, then test both against the last month of incidents. Reviewed: 2026-09-05 ## Source references - -