How should a binary protocol detect its version before it can trust the frame layout?

For a raw binary stream, use a small invariant envelope before decoding version-specific fields: stable magic bytes, an envelope version, bounded lengths, and fail-closed checks. TCP supplies a byte stream, so reads do not provide application-message boundaries (RFC 9293). A fixed prefix is a practical design recommendation for this case; an already negotiated transport protocol may provide the discriminator instead.

What must the receiver know before decoding a payload?

Define how the receiver recognizes the protocol, determines frame boundaries, and selects the payload schema. There is no universal order for learning these: a dedicated endpoint might already identify the protocol, while a negotiated connection might already select its version.

When multiple generations share an undifferentiated stream, avoid choosing a complete frame layout by guesswork. A corrupted byte must not become an unchecked allocation size or offset.

Give protobuf payloads explicit outer framing in this design. Its wire format consists of tagged fields with wire types; those types determine how each field’s payload is read (Protocol Buffers encoding guide). Field encoding and schema evolution do not replace your application’s framing contract.

What should the invariant envelope contain?

One proposed layout is:

MAGIC(4) | ENV_VER(1) | FLAGS(1) | HDR_LEN(2, big-endian) | BODY_LEN(4, big-endian) | TLVs | BODY

The numbers are byte widths. In this example, HDR_LEN includes the 12-byte fixed prefix plus optional metadata; BODY_LEN counts only body bytes. Specify these meanings in the wire contract, including minimum and maximum header sizes, maximum payload size, reserved flags, and supported envelope versions.

Magic helps detect wrong-protocol traffic or corruption; it does not authenticate a sender. A checksum can detect some accidental corruption, but it is not a substitute for cryptographic integrity.

Optional metadata can use explicitly defined type-length-value records after the fixed prefix. Protobuf is not uniformly encoded this way: its tagged wire format includes varints, fixed-width fields, and length-delimited fields. The guide calls the general scheme TLV, but not every protobuf field carries a separate explicit length.

Negotiate optional compression and capabilities only after establishing the base parsing rules. Define rejection behavior for unsupported versions instead of guessing an older layout.

How should the parser handle partial or malformed input?

Use explicit prefix, extended-header, and body states. Buffer until the required bytes are available, and handle multiple frames arriving in one read. Validate lengths before allocation, use checked arithmetic when combining them, and impose resource and time limits. TCP’s stream contract does not make one read equal one application frame.

For an ordinary request connection, closing on malformed input is a simpler default than attempting resynchronization. If recovery is a protocol requirement, bound the scan and require several checks: magic, supported version, valid flags, consistent lengths, and any defined integrity check. Finding magic inside payload data must not alone authorize a new frame.

How does this compare with gRPC framing?

Within gRPC over HTTP/2, each message has a five-byte prefix: a one-byte compression flag and a four-byte big-endian message length (gRPC over HTTP/2). These messages are carried in the HTTP/2 DATA stream; the prefix is neither the HTTP/2 frame header nor a standalone protocol-version detector.

The useful analogy is separating message boundaries from payload decoding. It does not mean copying gRPC’s five bytes solves your protocol’s version negotiation.

Which rollout checks matter?

Test supported old/new sender-receiver pairs, truncated prefixes, concatenated frames, oversized lengths, and unknown critical flags. Record unsupported-version failures and retire an older generation only against explicit compatibility criteria.

Should version be part of magic? Keep them separate in this proposed design for clearer diagnostics; other established formats may choose differently.

Can the header be variable-length? Yes, when a stable prefix or previously negotiated rule tells the parser how to locate and bound it.

Next, implement the parser states and fuzz their transitions before freezing the wire contract.

Reviewed: 2026-09-06.