Where should newline, BOM, and Unicode normalization happen in a file ingestion pipeline?

Decode bytes exactly once at the ingestion boundary, and make that boundary own BOM policy, charset validation, and an explicit newline policy. After that point, downstream code should see valid text under that contract or a hard failure. Preserve field content when the format permits meaningful embedded newlines. Unicode normalization is different: do not apply it globally during byte decode. Apply it only at the field or comparison boundary where your application has explicitly decided that canonically equivalent text should compare equal.

What should the ingestion boundary own?

Treat upload handling as a fixed sequence: raw bytes, declared media type and charset, byte-to-text decode, BOM handling, newline policy, format parse, then business validation. The important rule is ordering. BOM handling belongs with decoding because a BOM is a byte-stream concern, not a CSV or JSON parser quirk. RFC 3629: UTF-8 defines UTF-8 at the byte level and discusses the byte order mark, so this is the right layer to decide whether a given format accepts a BOM, rejects it, or strips it before exposing text.

Apply global newline normalization after decoding only when the format contract permits changing every newline, including newlines inside fields. For CSV, let the parser recognize record delimiters while preserving quoted field content unless the ingestion contract explicitly allows those field values to change. Record separation and embedded text are different concerns.

Where should Unicode normalization happen?

Not in the universal ingestion layer. Unicode Normalization Forms, UAX #15 explains canonical and compatibility equivalence and why normalized forms can give equivalent strings a unique binary representation. That still does not mean every text field should be rewritten on arrival. Normalizing the whole file can break signatures, hashes, forensic replay, or any workflow that must preserve original bytes.

Instead, define normalization per field or comparison rule. Example: if customer_id is a human-entered identifier and your product requirement says canonically equivalent forms must compare equal, normalize that field to NFC before uniqueness checks and persistence. If notes is free-form content, preserve exactly what the sender wrote. This fits RFC 8259, which defines JSON strings as sequences of Unicode characters and discusses string comparison separately rather than assigning one universal comparison rule.

How should CSV and JSON uploads behave?

For CSV: accept UTF-8, optionally accept a configured legacy encoding such as Windows-1252, and reject undecodable byte sequences. Do not silently replace bad bytes with �; the WHATWG Encoding Standard explains that decoding differences can change interpretation and recommends UTF-8 as the most appropriate encoding for interchange. If your CSV contract accepts a UTF-8 BOM, strip it during decode and record that fact; otherwise reject it at the boundary. For mixed line endings, use the contract’s accepted record-delimiter rules in the CSV parser. Do not rewrite quoted field newlines globally unless the contract explicitly permits that transformation.

Worked example:

• Input bytes: UTF-8 BOM + header row + mixed CRLF/LF
• Result: accepted if the endpoint allows a BOM and those record delimiters; BOM removed at the boundary, quoted field content preserved, original bytes retained for audit
• Fixture: a quoted multiline notes value containing CRLF
• Result: the parsed field retains those embedded characters unless its contract explicitly permits normalization
• Input bytes: claimed UTF-8 but invalid sequence
• Result: reject before CSV parsing and, if your decoder exposes it, report the byte offset and declared charset

For JSON exchanged across systems, require UTF-8 for interoperability as recommended by RFC 8259. In this ingestion architecture, prefer surfacing UTF-8 decoding failures at one boundary before or as part of JSON parsing, rather than letting each downstream parser handle encoding differently. Do not run global Unicode normalization before JSON parsing; JSON structure is text, but business meaning belongs above the parser. If producers send canonically equivalent names or values, normalize only the application fields whose equality semantics require it.

What should failures and observability look like?

Keep the original byte blob, the detected or declared encoding, whether a BOM was present, and whether newline canonicalization ran. Emit errors from the earliest failing stage: decode error before parse error, parse error before business-rule error. Also keep byte offsets where feasible and, once decoding succeeds, character positions relevant to parser or validation errors. That makes reprocessing deterministic and avoids parser-specific BOM hacks later.

Decision checklist:

• Which encodings are accepted for this endpoint?
• Is BOM accepted, forbidden, or stripped per format contract?
• Which record delimiters are accepted, and does the contract permit changing newlines inside quoted fields?
• Which fields, if any, are normalized to NFC or NFKC?
• Which fields must preserve exact original text?
• Are invalid byte sequences rejected rather than replaced?

Q: Should I normalize every string to NFKC for safety? A: No. Unicode Normalization Forms, UAX #15 distinguishes canonical equivalence from compatibility equivalence and explains that compatibility-equivalent forms can preserve distinctions that matter in some contexts. Reserve NFKC for narrowly defined identifiers when that equivalence is a product requirement.

Q: Can the CSV parser just ignore BOMs itself? A: It can, but that scatters byte-level policy across parsers. Put BOM handling in the single decode boundary so every ingestion path behaves the same.

Next step: write this contract as a shared ingestion spec and test-fixture set, then run every upload path against the same BOM, newline, invalid-byte, and Unicode-equivalence cases.

Reviewed: 2026-09-06.