Change an Event Schema Without Stopping the Stream

Plan schema changes across producers, consumers, and replay

A registry compatibility check covers structural rules, while safe evolution also needs deployment order, semantics, and replay tests. Inventory oldest retained event, oldest supported consumer, and every producer before selecting compatibility mode. Confluent schema evolution supplies the primary specification or operating model for this decision.

Add a currency field without breaking old consumers

To add currency, deploy consumers that tolerate absence first, then producers. A rename should add a new field and keep the old one through the retention window. Tombstones and changed units require explicit semantic versions or new event types. CloudEvents Specification provides the complementary protocol or operational detail.

Deploy in compatibility order

For an optional currency field, release consumers that tolerate absence before producers emit it. If older consumers cannot ignore unknown fields, fix or retire them first. Renaming requires an overlap: emit both old and new fields until every supported consumer and the retained replay window no longer need the old name.

Structural registry checks cannot detect semantic changes. Reusing amount with a new unit or changing timestamp meaning may pass the schema while corrupting projections. Use a new field or event type and document conversion.

Rehearse historical replay

Keep sanitized samples from the oldest retained schema and important edge cases. In CI, deserialize them with the new consumer and compare durable outcomes, not byte-for-byte internal objects. Record producer version and stable envelope metadata so failures can be traced.

Test unknown fields, missing optional fields, duplicate events, out-of-order entity versions, tombstones, and a consumer that restarts from an old offset. Inventory the oldest event and oldest supported consumer before choosing a registry compatibility mode. For the next change, write the producer-consumer deployment sequence and retirement condition beside the schema change; “registry accepts it” is only the first gate. Measure replay time with production-like volume; compatibility is incomplete if recovery exceeds the retention or incident objective.