What evidence should we capture before restarting a failing service?

If restarting will erase the only useful clues, take a short, fixed evidence budget before you do it—but only after you decide the mitigation path and the maximum delay you will tolerate. That ordering boundary matters. For a transient hang, capture the current telemetry, recent logs, instance identity, recent changes, dependency state, and one targeted runtime snapshot, then restart or fail over on time.

Google’s SRE guidance says to look at “the system’s telemetry and logs” to understand current state, and incident response guidance defines success as mitigating impact and restoring service in a structured way with a working record (Effective Troubleshooting, Incident Response).

When should you capture evidence before restarting?

First choose the corrective action and set a hard time budget for delay. That is the enforced boundary: do not start collecting diagnostics until you know whether you will restart, fail over, drain traffic, or replace the instance, and how long evidence capture is allowed to postpone that action.

If the service is causing broad user impact, exhausting shared capacity, or contributing to cascading failure, skip capture and mitigate immediately. The SRE incident response model is clear that resolving an incident means “mitigating the impact and/or restoring the service,” not waiting for full diagnosis first (Incident Response).

What should you collect in a 1–5 minute budget?

Capture the highest-value state that a restart will destroy:

• incident basics: timestamp, service name, pod or host ID, alert name, symptom summary
• current telemetry: error rate, latency, saturation, queue depth, traffic, restart count
• recent local logs from the failing instance
• dependency state: database, cache, broker, upstream availability or timeout signals
• change markers: deployed version, config revision, feature flag state, recent rollout
• one runtime-specific snapshot for hangs, such as a thread dump, goroutine dump, or stack sample

This matches the troubleshooting flow in the SRE book: start from observations, then test likely causes against what the system is doing now (Effective Troubleshooting).

Avoid defaulting to heavyweight capture. Full heap dumps, packet captures, and repeated profilers can worsen an already unhealthy instance.

When do you stop collecting and restart now?

Stop immediately when the budget expires, latency or error rate worsens, CPU or disk pressure climbs, or responders lose confidence that the instance can stay in service safely. Also stop once you have enough to answer the first routing question: is the process internally stuck, or is it blocked by a dependency?

Keep a concise incident record while you do this. The incident response workbook explicitly says to “Keep a working record of debugging and mitigation as you go” (Incident Response).

How should you store the artifacts safely?

Assume dumps and logs may contain secrets, tokens, request bodies, or personal data. Store raw artifacts in restricted incident storage with limited access, retention rules, and a ticket or incident reference. Put summaries and links in the shared incident timeline, not the raw files themselves.

What does a practical checklist look like?

Example: one API pod is hung, other replicas are healthy.

1. Decide: drain and restart this pod in three minutes unless impact expands.
2. Record time, pod ID, customer symptom, and mitigation plan.
3. Save the last 15 minutes of service and dependency graphs.
4. Export recent pod logs.
5. Capture one stack dump from the runtime.
6. Note image version, config revision, and latest rollout event.
7. Drain or restart at the timer, even if analysis is unfinished.
8. Hand artifacts to the service owner for post-incident review.

What if collection itself increases saturation? Stop collection and mitigate immediately. Recovery takes precedence over preserving clues.

Who should investigate the retained evidence later? The service owner or explicitly assigned follow-up investigator should review it after stabilization, using the incident timeline to align artifacts with events.

Your next step: add a per-service pre-restart evidence budget, stop conditions, and exact runtime commands to the runbook before the next incident.

Reviewed: 2026-09-05