# Treat Runbooks as Perishable Production Dependencies > Keep emergency procedures executable by testing prerequisites, permissions, mutations, stops, and rollback on a cadence. Canonical URL: https://www.devobs.io/articles/runbook-verification-and-decay/ By: Ines Costa Published: 2024-10-11T15:12:26.347Z Updated: 2026-09-06T10:18:15.722Z Section: Architecture ## Treat every runbook step as a production dependency A runbook is valid only while its commands, permissions, dashboards, targets, and expected outputs match production. Give each step prerequisites, a read-only diagnostic, bounded mutation, stop condition, verification, and rollback. [AWS Systems Manager runbooks](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-documents.html) supplies the primary specification or operating model for this decision. ## Rehearse database failover prerequisites A database failover runbook may fail because the role lost permission or the dashboard name changed. Execute diagnostics monthly in production safely, rehearse mutations in a representative environment, and timestamp the last successful verification per step. [Kubernetes debugging guide](https://kubernetes.io/docs/tasks/debug/debug-application/) provides the complementary protocol or operational detail. ## Write decisions around observable signals A useful runbook begins with user impact and the alert condition. Each step states what to inspect, the expected result, what an alternative result means, and the next branch. Commands include scope, prerequisites, and safe output. Avoid copying secrets or broad destructive commands into the page. Link dashboards and automation by stable identifiers. Record the service revision, required role, dependency version, and last successful exercise. If access or topology changes, the runbook should become due for review automatically. ## Test with a reader, not the author Give the scenario to an on-call engineer who did not write the procedure. Inject a safe failure and observe where they pause, guess, or leave the documented path. Measure time to diagnosis and mitigation, then update the exact ambiguous step. Verify read-only diagnosis first and make any mutation require a stated rollback or confirmation signal. An automated document still needs tested parameters, timeouts, and partial-failure behavior. Select the runbook attached to the highest-severity alert and execute it in staging this week; record owner, date, outcome, and the revision that invalidates the result. Also validate the escalation target and production permissions; a technically correct command is useless when the responder cannot reach the system or owner. ## Source references - -