How should an alert distinguish zero errors from missing metrics?

Zero errors and missing telemetry should be different alert states. Treat them as four separate conditions with different owners: healthy zero means you observed requests and saw no errors; absent series means the application stopped emitting a metric; failed scrape means monitoring could not collect it; intentionally retired target means the workload was removed on purpose. In Prometheus, use normal error-rate queries for symptoms, absent() or absent_over_time() for missing series, and separate metamonitoring alerts for scrape health so telemetry gaps cannot silently look like recovery.

What states should the alert model?

Model four states explicitly.

Healthy zero: traffic exists and the error metric is zero. This is the only case that should read as “no errors.”

Absent series: the metric name or label set is gone. Prometheus documents absent() as useful for alerting when no time series exist, and absent_over_time() when none exist for a period. This usually belongs to the service owner or instrumentation owner.

Failed scrape: Prometheus cannot collect from the target or a telemetry path is broken. That is not application health; it is monitoring health. Prometheus explicitly recommends metamonitoring so you have confidence the monitoring system is working.

Intentionally retired target: the service, shard, or region was decommissioned. This should suppress the absent-series page only when your source of truth says the target is gone on purpose.

Why does a missing denominator break error-rate alerts?

A common failure mode is an alert like:

sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m])) > 0.05

Prerequisite: http_requests_total must be a scraped counter with a code label on the same targets in both numerator and denominator.

If scrape loss removes samples from both numerator and denominator for part of the target set, aggregated ratios can become misleading or disappear, depending on label scope and rule construction. The important operational point is simpler than the exact expression outcome: missing telemetry is not evidence of zero errors.

For alerting on error rates, avoid masking missing telemetry by defaulting absent series to zero with or vector(0). Instead, pair the symptom alert with a telemetry-presence alert such as:

absent_over_time(http_requests_total{job="api"}[10m])

and a separate scrape-health alert for the target set you expect to be monitored. This matches Prometheus guidance to keep alerting simple, alert on symptoms, and use metamonitoring to confirm the monitoring system is working.

How should retired targets and stale data be handled?

Do not synthesize healthy samples for deleted workloads. Instead, join your alert scope to an inventory that defines which targets are expected right now: active services, active regions, or scheduled jobs. If a target disappears but is still expected, page on missing telemetry. If it is no longer expected, resolve the telemetry alert by removing it from expected inventory, not by forcing zero values.

A practical checklist:

• Page on user-visible error rate from observed traffic.
• Alert separately when a required metric series is absent for long enough to outlast rollout churn.
• Alert separately when scraping or the monitoring pipeline is unhealthy.
• Suppress missing-series alerts only from an authoritative retirement signal.
• Scope all three alerts by region or shard so partial loss stays visible.

What if only one region disappears?

Yes, alert per region. Otherwise global aggregation can hide a missing region behind healthy traffic elsewhere. Run both the symptom alert and the absent-series alert with region in scope.

When can a retired target stop paging?

When the same system that creates expected targets also marks that target retired. If deletion is manual, keep paging until inventory changes; otherwise missing metrics can become a silent failure mode.

Next step: audit one existing error-rate alert and add one paired absent_over_time() rule plus one scrape-health rule for the same label scope.

Reviewed: 2026-09-06.