# When is an incident resolved if errors are gone but a backlog remains?

> Use separate states for mitigation, user-visible recovery, and closure in asynchronous systems.

Canonical URL: https://www.devobs.io/articles/qa-incident-recovery-closure-criteria/
By: Lena Fischer
Published: 2025-02-23T04:50:58.181Z
Updated: 2026-09-06T08:31:04.426Z
Section: Architecture

If errors have stopped but queued work is still delayed, the incident is usually mitigated, not fully resolved. In asynchronous systems, close the incident when customer impact is no longer actively worsening, the remaining backlog has explicit ownership, and you have evidence for oldest affected work, drain expectations, and data reconciliation. Do not wait for an arbitrary quiet period. Do not declare resolution just because dashboards are green again. Google’s SRE guidance defines resolving an incident as mitigating impact and/or restoring service, which is broader than clearing one error signal ([Google SRE Workbook: Incident Response](https://sre.google/workbook/incident-response/)).

## What counts as resolved versus mitigated?

Treat these as separate states.

**Mitigated** means the triggering fault is contained: new work is being accepted again, error rates have returned to normal, or the failing dependency is stable. **User-visible recovery** means customers are again receiving outcomes within acceptable delay bounds. **Closed** means the active incident response can stand down because the remaining work is understood, measured, assigned, and no longer requires cross-functional incident coordination.

That distinction fits Google’s incident framing: incident work is about both restoring service and coordinating responders effectively ([Managing Incidents](https://sre.google/sre-book/managing-incidents/)). If backlog drain still needs an [incident commander](https://www.devobs.io/articles/incident-command-for-small-teams/), frequent stakeholder updates, or cross-team operational decisions, you are not done.

## Can you close while backlog remains?

Yes, if the backlog is now a controlled recovery task rather than an active incident.

Use closure criteria tied to customer obligations, not just system health:

- New incoming work succeeds at normal rates.
- The backlog is shrinking at a measured rate.
- You know the **oldest affected item** and its current age.
- You have checked for data loss, duplication, or ordering damage where that matters.
- One team and one owner are accountable for recovery after handoff.
- Reopen conditions are written down before standing down.

A quiet period is a poor rule for asynchronous systems because queues can hide impact after request errors disappear. Backlog age is usually the better signal than raw queue depth: 5,000 items may be fine in one system and severe in another, but “oldest job is 3 hours late” is immediately meaningful.

## How should this work in practice?

Example: a job ingestion service returned 500s for 20 minutes. You rolled back the bad deploy. API errors are gone, but 180,000 customer jobs remain queued.

A solid incident timeline looks like this:

1. **Mitigated at 14:20**: rollback complete, new submissions succeed.
2. **Recovered for most users at 15:10**: backlog age falls from 95 minutes to 12 minutes, which is within your published delay tolerance.
3. **Incident closed at 15:30**: remaining backlog drain is owned by batch-platform ops; oldest affected job ID range is identified; replay and reconciliation checks are running; reopen if backlog age rises for 15 minutes or reconciliation finds missing jobs.

That closure is defensible because you are not pretending all work is finished. You are saying active incident command is no longer needed.

## What evidence should the incident commander require before closure?

Use this checklist:

- Current backlog size and oldest-item age
- Estimated drain rate and next decision point
- Result of integrity checks or a scheduled reconciliation job
- Named recovery owner after the incident bridge ends
- Customer-facing status: recovered, delayed, or partial
- Explicit reopen triggers

This follows the spirit of structured incident management: clear roles, a working record, and coordinated handoff rather than ambiguous “looks good now” decisions ([Google SRE Workbook: Incident Response](https://sre.google/workbook/incident-response/)).

## Follow-up: Who owns overnight backlog drain?

Not the incident commander by default. Once closure criteria are met, assign recovery to the service owner or operations team with a concrete handoff note, metrics to watch, and pager thresholds for reopening.

## Follow-up: When should the incident reopen?

Reopen if customer-visible delay stops improving, backlog age breaches the agreed threshold again, or reconciliation finds missing, duplicated, or corrupted work. A green error-rate chart does not overrule failed recovery evidence.

Next step: add separate incident states for mitigated, service recovered, and closed to your [runbook](https://www.devobs.io/articles/runbook-verification-and-decay/), and require oldest-backlog-age plus reconciliation status in every closure note.

Reviewed: 2026-09-05.

## Source references

- <https://sre.google/workbook/incident-response/>
- <https://sre.google/sre-book/managing-incidents/>