# How do we keep model input drift from being introduced by our own serving pipeline? > Keep serving pipelines from introducing input drift by versioning the feature contract, validating assembled features at inference, and checking train-serve parity before cutover. Canonical URL: https://www.devobs.io/articles/qa-ge50-ml-input-contract-drift/ By: Lucas Vale Published: 2025-12-23T17:26:05.906Z Updated: 2026-09-06T10:18:15.722Z Section: Architecture Treat self-inflicted input drift as a contract failure between training and inference, not just a monitoring symptom. The practical pattern is to version the model input contract separately from the model, validate requests and assembled features against that contract, and verify train-serve parity during rollout. If the serving path cannot prove equivalent schema, defaults, and time semantics, it should reject the prediction or enter a defined degraded path rather than silently scoring. ## What belongs in the input contract? The contract should define feature names, types, units, nullability, allowed ranges, default behavior, categorical vocabulary handling, and time semantics such as event time versus processing time. [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv) supports schema-based validation, lets you codify expectations such as data types and categorical values in a schema, and documents training-serving skew detection. Complete feature assembly under a declared contract version before model inference starts. That does not require atomic transactions across unrelated services. It does require that each prediction record the model version, contract version, and the source timestamps used to build time-dependent features. ## Where should feature logic live? There are three workable patterns. A shared library gives the strongest parity because training and serving reuse the same implementation. The tradeoff is tighter release coupling. A feature service or materialized feature view improves operational control and makes freshness rules explicit, but it adds a runtime dependency and forces you to define fallback behavior when data is unavailable. Duplicated logic can still work when offline and online stacks differ, but only if you continuously compare outputs. Google’s [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml), in Rule #5, recommends testing infrastructure separately from machine learning behavior. That is the right posture for train-serve parity. ## Which serving changes usually create drift? The common failures are boring software changes: a missing-value default changes from `0` to null, one path clips a feature and the other does not, training uses event time while serving uses wall-clock time, or a categorical vocabulary changes without a version bump. Backfilled schemas are another trap: offline training starts depending on a field that serving still treats as optional. Worked example: a fraud model consumes `account_age_days`, `country_code`, and `chargebacks_30d`. Training computes `account_age_days` from account creation event time. Serving accidentally recomputes it from request receipt time after a service refactor. Types still match, but semantics do not. The fix is to encode the time basis in the contract, log the timestamp source with each prediction, and shadow-compute both old and new feature vectors on sampled traffic before cutover. ## How do we enforce and verify the contract? At ingress, validate raw request shape, required identifiers, and timestamp presence. After feature assembly, validate the final inference vector: required features present, ranges valid, vocabulary version matched, and serving-only defaults explicitly applied. [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv) also documents schema environments, which are useful when training legitimately has labels that serving should not. Use this rollout checklist: - Log model version and feature-contract version on every prediction. - Shadow-compute new features beside the current path. - Recompute sampled online requests offline and compare values. - Alert on missingness, range violations, and vocabulary mismatches. - Keep golden exemplars for regression tests. - Roll back feature logic independently from model weights. ## How do we evolve safely when features change? Introduce new feature versions additively. Let models declare which contract versions they accept, run both versions during a canary, then remove the old one only after parity checks and distribution checks stay clean. Keep incident response simple: if parity breaks, stop the new feature path first, then decide whether the model should also roll back. ## Follow-up Q&A? **Should invalid inputs be rejected or filled with defaults?** Reject by default for high-impact decisions. Use defaults only when the contract defines them explicitly and you have tested their effect. **How do we tell pipeline drift from real-world drift?** Replay the same raw events through offline recomputation. If offline and online vectors differ, it is pipeline drift. If they match and the distribution still moved, the world changed. Next step: pick one production model, write its feature contract into version control this week, and add one parity test that compares serving features with offline recomputation on stored exemplars. Reviewed: 2026-09-06 ## Source references - -