When should recurring on-call work become an engineering project instead of a runbook step?

Recurring on-call work should become an engineering project when it is no longer mainly diagnosis and judgment, but repeated, interrupt-driven, preventable work that keeps consuming operator minutes. Use four triggers together: recurrence, hands-on time, interruption cost, and a plausible removal path. A runbook tells responders how to survive the next occurrence; a project exists to make that occurrence rarer, shorter, or gone.

Is this work still incident response, or is it toil?

Google’s SRE definition is still the useful boundary: toil is work tied to running a service that is manual, repetitive, automatable, tactical, and lacking enduring value (Google SRE book: Eliminating Toil).

If the responder must interpret ambiguous signals, weigh business tradeoffs, or coordinate multiple teams, keep investing in runbooks, dashboards, and training. But if the responder is reliably following a sequence like “check lag, restart worker, replay jobs, verify recovery,” the work has crossed into repeatable toil. The SRE Workbook explicitly notes that runbook steps such as “log in, execute this command, check the output, restart Y” are “essentially pseudocode” and strong candidates for automation or design fixes (Google SRE Workbook: Eliminating Toil).

What trigger should move it into project work?

Do not use frequency alone. Promote the work when all four of these are true:

• Recurring: you have seen the same failure mode enough times to describe it clearly. Google distinguishes first-time and second-time work from toil; toil is work done “over and over” (Google SRE book: Eliminating Toil).
• Costly in operator time: measure hands-on minutes, even if a script is involved. The book is explicit that manually running a script still counts as toil (Google SRE book: Eliminating Toil).
• Interrupting engineering work: repetitive alerts drain attention and can hide more important signals (Google SRE Workbook: Eliminating Toil).
• Preventable: you can name a removal mechanism: automate remediation, remove the trigger, redesign the component, or fix the root cause.

That last point is the real gate. If you cannot state how the work disappears, you probably need more diagnosis before you need a project.

How should you scope the project and prove it is done?

Assign one owner and one removal metric.

Worked example: a team receives 6 pages in 30 days for a stuck ingestion consumer. Each page takes 20 hands-on minutes: inspect backlog, restart one worker set, replay failed items, confirm recovery. This is deterministic, repeated, and interrupt-driven. It should be project work.

A good project statement is: Eliminate pages from consumer deadlock by fixing lock handling and adding automatic replacement for unhealthy workers.

Use an exit criterion like this:

• zero pages for this failure mode for 30 days, or
• manual recovery reduced from 20 minutes to under 5 minutes

The ordering boundary matters: first ship the change that removes or automates the repeated repair, then demote the old runbook steps. A precheck or helper script is not success if the same page still demands the same human sequence.

Short checklist:

1. Identify one repeated failure mode.
2. Count monthly occurrences.
3. Estimate hands-on minutes per occurrence.
4. Estimate interruption cost to planned work.
5. Name the removal mechanism.
6. Assign an owner.
7. Define the exit metric before starting.

When should it stay a runbook step?

Keep it as a runbook step when human judgment is still essential, or when automation would create more maintenance than it removes. The Workbook warns that workaround scripts can mask the real issue; sometimes the right investment is root-cause removal, not more operational glue (Google SRE Workbook: Eliminating Toil).

What if automation itself becomes toil?
Prefer the smallest durable change. If the automation needs frequent babysitting, redesign or eliminate the trigger instead.

Who should fund work shared across services?
Give ownership to the team that can remove the cause, and make the saved interrupt time visible across affected teams.

Next step: review the last 30 days of pages, group them by repeated failure mode, and open one engineering project for the pattern with the clearest removal path and the highest hands-on cost.

Reviewed: 2026-09-05.