# When does a nested schema change in object storage stop being backward-compatible?

> A practical framework for deciding when nested schema evolution remains safe across file formats, table metadata, and reader behavior in object storage systems.

Canonical URL: https://www.devobs.io/articles/qa-ge50-when-does-a-nested-schema-change-in-object-storage-stop-being-backward-compatible/
By: Finn Larson
Published: 2023-04-26T18:20:45.936Z
Updated: 2026-09-06T08:31:04.426Z
Section: Architecture

A nested schema change stops being backward-compatible when a supported reader can no longer read old and new files with the same meaning from the same table contract. The practical boundary has three layers: file format, table metadata, and engine behavior. A change may be representable in Parquet or allowed by table metadata, yet still be unsafe if readers interpret fields differently, surface historical gaps as nulls, or require a coordinated cutover to preserve semantics.

## What defines compatibility in object storage?

[Parquet’s types documentation](https://parquet.apache.org/docs/file-format/types/) says the file format’s supported types are intended to be “as minimal as possible,” with a focus on “on disk storage.” Because that scope is deliberately narrow, you should not infer full dataset-contract compatibility across engines from file schemas alone.

Table formats add stronger rules and publication semantics. The [Apache Iceberg specification](https://iceberg.apache.org/spec/) says Iceberg tables support full schema and partition spec evolution, that schema evolution supports safe column add, drop, reorder, and rename including in nested structures, and that older readers may not correctly read newer table features when a newer format version introduces forward-incompatible changes. The same specification also describes committed snapshots, explicit commits, atomic metadata swaps, and optimistic concurrency control ([Apache Iceberg specification](https://iceberg.apache.org/spec/)). That gives you the right test: a nested change is backward-compatible only if every reader version you still support understands both the metadata change and the mixed population of historical and newly written files.

## Which nested changes stay compatible?

Adding an optional field inside a struct is often the safest kind of in-place change, but you still need to verify reader behavior. In [Delta Lake’s schema evolution article](https://delta.io/blog/2023-02-08-delta-lake-schema-evolution/), old rows show `null` for newly added columns after schema evolution is enabled. Treat that as a demonstrated pattern for Delta’s added-column example, not as a universal guarantee for every nested-field change across engines.

The [Apache Iceberg specification](https://iceberg.apache.org/spec/) also states that schema evolution supports reorder and rename, including in nested structures. Even so, treat reordering as operationally risky until you verify that every engine and downstream consumer you support actually honors table metadata rather than hidden positional assumptions in application code, extracts, or BI models.

Do not assume primitive widening is safe just because it seems intuitive at the file level. Verify the exact promotion rules in the table-format and engine documentation you actually run before treating that change as backward-compatible.

## Which changes cross the boundary?

In practice, treat changing an optional nested field to required as a contract break unless you have a documented migration path, such as rewriting historical data or enforcing a default outside the stored historical rows. Old files do not automatically satisfy the stronger contract.

Renaming a nested field is only safe if every consumer follows field identity through table metadata. Iceberg can represent renames in schema evolution metadata ([Apache Iceberg specification](https://iceberg.apache.org/spec/)), but many real consumers still bind to field names in SQL text, serializers, dashboards, and extraction code. For a live table, a rename often behaves like a contract break even when the table format can represent it.

We recommend treating array or map shape changes, and replacing one struct with a different nested shape, as a new dataset version. Those changes often change how readers and downstream code interpret existing values, which is a poor fit for in-place evolution.

## Where do silent failures come from?

The dangerous case is permissive evolution. The [Delta Lake schema evolution article](https://delta.io/blog/2023-02-08-delta-lake-schema-evolution/) shows that when schema evolution is enabled, existing rows can get `null` for added columns, appends with fewer columns can succeed, and even appends with no schema overlap can still be accepted. The same article explicitly calls this behavior “rather permissive” and warns that schema evolution can break downstream processes ([Delta Lake schema evolution article](https://delta.io/blog/2023-02-08-delta-lake-schema-evolution/)). A successful write is not proof of compatibility.

Worked example: suppose `events` contains `context.app.version` as a string, and you want `context.app.build` as an integer plus `context.app.channel`.

Safe rollout: add optional `channel`, keep `version`, publish `build` through a derived compatibility view or a rewritten target table, then cut consumers over at a metadata or snapshot boundary in snapshot-based table formats after validation reads confirm old and new data behave as expected.

Unsafe rollout: rename `version` to `build` and change string to integer in one commit. This combination is high risk: depending on reader and metadata handling, consumers may fail, return nulls for missing fields, or interpret old and new rows differently.

## What is the practical decision rule?

Allow in-place evolution for optional nested adds when your table format and readers document that behavior clearly. Be much more conservative with renames, reorderings, and type changes, because the file format, metadata layer, and engine may not fail in the same way.

For snapshot-based table formats such as Iceberg, the snapshot commit is the practical publication boundary readers observe. The [Apache Iceberg specification](https://iceberg.apache.org/spec/) describes explicit commits, atomic metadata swaps, committed snapshots, and optimistic concurrency, and it also notes that writers can select what to validate and make different isolation guarantees. Checks that run earlier can be invalidated by concurrent writers before commit. Before publishing a new contract, run every supported engine against historical files and the new snapshot, compare row counts and null rates, and cut over at the metadata commit that defines the contract your readers will consume.

**Can nulls for old rows still be backward-compatible?**

Yes, but only if null is already the agreed meaning for “field absent in historical data” across all supported readers and downstream logic.

**Should I rely on a pre-append schema check?**

Not by itself. In systems such as Iceberg, visibility changes at commit time, and pre-commit checks can be invalidated by concurrent writers. Use pre-append checks as an early guardrail, then validate again as close as possible to the publish boundary.

Your next step is to write down the exact reader versions and downstream contracts you support, then test the proposed nested change against that matrix before deciding whether to evolve in place, rewrite, or version the dataset.

Reviewed: 2026-09-06

## Source references

- <https://parquet.apache.org/docs/file-format/types/>
- <https://iceberg.apache.org/spec/>
- <https://delta.io/blog/2023-02-08-delta-lake-schema-evolution/>