Design Releases So Rollback Survives Schema Changes

Reviewed: 2026-09-06

Keep old and new binaries compatible

Old and new binaries overlap during a rolling update, so both must understand the live schema and configuration. Expand before contract: add nullable fields, dual-write, backfill, switch readers, and drop old fields only in a later release. Kubernetes Deployments illustrate the overlap problem during rolling updates: the controller gradually scales up a new ReplicaSet while scaling down the old one at a controlled rate, and it can roll back Deployment revisions if the rollout is not stable.

Test a name-schema migration across rollback

Replacing full_name with structured names requires old-new-old compatibility tests. An external payment or notification is already a side effect, so after that point recovery means compensation or roll-forward. GitHub’s continuous-deployment documentation shows how teams can automate build, test, approvals, and deployment controls in a CD workflow, but those controls do not by themselves make schema and side-effect recovery safe.

Expand before you switch

Deploy additive schema and protocol changes first: nullable columns, dual readers, versioned event fields, and cache formats old code can ignore. New code should continue reading the old representation during the overlap. Backfill in bounded batches with checkpoints, then switch writes only after both versions are safe.

A deployment controller can restore an earlier image, but it cannot reverse a dropped column, a sent event, or an external charge. Mark the point of no return and separate image rollback from data compensation.

Contract after evidence

Before removing the old path, prove no supported process emits or consumes it. Check deployment inventory, job workers, scheduled tasks, mobile clients, and replay consumers. Gate cleanup on telemetry rather than the date the new release began.

Exercise three moments: failure before the write switch, failure after dual writes begin, and failure after a forward-only migration. Verify the documented response for each. Keep event and cache format versions until their retention windows close.

Include data repair time in the release risk. A five-minute image rollback can still leave hours of records requiring version-aware reconciliation.

The practical next step is simple: for your next release, list every state change the new version touches—schema, data backfills, cache keys, emitted events, and external calls—and mark which ones are reversible, compensatable, or strictly forward-only before you ship.