Use shadow reads to prove a migration before cutover

Shadow reads can prove that a new datastore or service behaves like the old path under real traffic, provided the shadow path cannot affect the user and mismatches are interpreted against a written contract. Mirror selected reads, suppress every side effect, compare normalized results asynchronously, and define cutover thresholds before looking at the numbers.

Mirror an operation, not raw traffic blindly

Copy the logical request after authentication and validation, then send it to the candidate system with an explicit shadow marker and a short deadline. Envoy’s request mirroring documentation describes fire-and-forget shadowing in which the mirrored response is discarded. That is useful for load exposure, but a migration comparison also needs controlled capture of normalized results. Instrument that comparison in a component that cannot alter the primary response.

Use a read-only credential and block writes at network, API, and datastore layers. Watch for hidden effects: cache population, lazy migrations, access-time updates, analytics events, rate-limit counters, and calls to downstream services. A method named GET is not proof of purity. Build a test that fails if the shadow identity attempts a mutating operation.

Sample deterministically by tenant or resource so repeated requests stay comparable and privacy rules remain enforceable. Exclude requests containing data the new environment is not permitted to receive. Redact comparison output and store only the fields needed to diagnose mismatches.

Normalize what the contract says is equivalent

Raw byte equality produces noise. JSON object key order, timestamps, generated links, trace IDs, and nondeterministic list order may differ while behavior remains equivalent. Define canonicalization per endpoint: sort only collections whose public contract is unordered, round timestamps only to promised precision, and ignore fields explicitly documented as unstable. Do not normalize away a difference merely because it is common.

Classify mismatches as missing record, extra record, value difference, ordering difference, error-class difference, latency timeout, or comparison failure. Attach primary and candidate version, request shape hash, tenant cohort, and trace correlation. OpenTelemetry’s overview of distributed traces explains traces as paths through distributed systems composed of spans; linked or correlated spans help determine whether a mismatch originated in the new service or one of its dependencies.

Reconcile against risk, not one percentage

A 99.99 percent match rate can hide every administrator request failing. Define thresholds by operation and consequence. Any authorization leak, missing financial record, or incorrect write-precondition result is a stop condition. A formatting difference may be accepted after the public contract is confirmed. Require a minimum sample for critical cohorts, not only a fleet-wide total.

Track mismatch age and recurrence. A transient replication-lag mismatch should converge within a known window; one that persists indicates data divergence. Re-run comparisons after backfills and schema changes, but preserve the earlier classifications so progress cannot erase evidence.

Make cutover reversible

Before cutover, require zero unresolved critical classes, stable rates through peak and maintenance periods, and completed reconciliation for sampled mismatches. Route a small read cohort to the new path with the old system still available, then increase exposure. Keep a rollback trigger tied to user-visible errors and divergence.

Start with one side-effect-free endpoint. Write its equivalence rules and stop conditions, mirror one percent of eligible tenants, and manually classify the first fifty mismatches. That exercise will reveal whether the comparison is measuring contract differences or merely serialization noise.

Avoid loading the candidate beyond reality

Mirroring duplicates work, so cap shadow concurrency and drop samples before harming the primary path. Record drops as missing evidence rather than successful matches. Compare candidate latency separately from equality, including queue time introduced by the mirror. Rehearse a slow candidate, a full comparison queue, and a failing comparison store. A candidate that appears correct only because difficult requests were dropped or protected from realistic load is not ready for cutover.

Reviewed September 2026.