How do we detect that the paging system itself has stopped working?

Detect pager failure with a continuous synthetic alert that follows the same route as a real page, then make missing arrivals page someone through a different path. Do not rely on health checks for Prometheus, Alertmanager, or the paging vendor alone. The reliable design is end-to-end metamonitoring plus an independent silence detector. Prometheus explicitly recommends black-box testing of the alert path and using external monitoring as a fallback (Prometheus alerting practices).

What path should the heartbeat test?

Test the whole chain you expect during an incident: rule evaluation, routing, notification delivery, and human-visible receipt. If your production path is Prometheus to Alertmanager to a paging provider to push, SMS, or voice, the heartbeat should traverse that exact route.

That is better than checking each component in isolation. Prometheus recommends metamonitoring and gives the example that a black-box test of alerts flowing through the stack is better than separate alerts on each component (Prometheus alerting practices). Google’s SRE guidance makes the same distinction: white-box monitoring tells you about internals, while black-box monitoring tests externally visible behavior as a user would see it (Google SRE book, Monitoring Distributed Systems).

How should the silence detector be designed?

Start simple:

• send one synthetic page every 60 seconds
• record each successful arrival at an independent receiver
• fire a silence alert after 3 missed intervals, such as 4 minutes without arrival
• route that silence alert through a secondary path owned by a real on-call team

The receiver must be in a different failure domain. Use a separate cloud project or account, separate credentials, and ideally a separate monitoring stack. If the sender and checker share the same failure, you have not actually tested recoverability.

Keep this alert narrow. The only question is whether a known test notification arrived on time. That fits the SRE advice that pages should have good signal and very low noise, because paging humans is expensive (Google SRE book, Monitoring Distributed Systems).

What happens when a credential breaks?

Suppose Alertmanager uses an API token to call your paging provider. The token expires at 02:13.

At 02:14, the heartbeat still evaluates and routes correctly, but the provider rejects delivery. Your independent receiver does not observe the expected arrival. At 02:18, the silence detector crosses its threshold and triggers a secondary escalation path, such as a separate provider, a ticket queue with phone follow-up, or a manual call tree.

The incident is now clearly “paging pipeline failed,” not “application down.” The responder runbook should start with: confirm the missing heartbeat, inspect recent routing or credential changes, send a manual test notification, restore the primary path, and only then resolve.

How do maintenance windows and ownership work?

Maintenance should be explicit. If you pause the sender, the receiver must know the approved window and expected resume time; otherwise you create false positives.

Ownership should also be explicit: one team owns the synthetic sender, one owns the independent receiver, and one on-call rotation owns the secondary escalation path. If all three depend on the same broken system, redesign the path.

Follow-up: Who monitors the alerting pipeline?

Treat it like production infrastructure. Usually the platform or SRE team should own the heartbeat because they control routing, provider credentials, and escalation policy.

Follow-up: How do I test this when no real pages happen?

Do not wait for an outage. Run the synthetic page continuously, and periodically break a non-production notification credential on purpose to prove the silence detector and secondary route still work.

Reviewed: 2026-09-05