# How do I compare a replacement algorithm with the old implementation when neither is a perfect oracle? > Use the old and new implementations as disagreement detectors, not as judges of correctness. Canonical URL: https://www.devobs.io/articles/qa-differential-test-replacement-algorithm/ By: Nina Patel Published: 2024-01-17T05:48:28.223Z Updated: 2026-09-06T10:18:15.722Z Section: Architecture Use the old and new implementations as disagreement detectors, not as judges of correctness. Run both on the same inputs, normalize any irrelevant output differences, and treat every mismatch as a case to adjudicate against specifications, invariants, and a small curated set of independently checked examples. Property-based testing helps you generate broad input coverage, and coverage-guided fuzzing helps you drive into rare paths, but neither implementation should be treated as the oracle just because it is older. ## What should differential testing prove here? It should prove that the replacement is compatible where compatibility matters, and expose where the two versions disagree. That is exactly the useful framing from [Hypothesis equivalent testing](https://hypothesis.readthedocs.io/en/latest/reference/integrations.html), which describes differential testing as a case where “none of the compared functions are fully trusted but any difference indicates a bug.” The bug may be in the old code, the new code, or your assumptions. That means your test harness should record three things per input: the normalized outputs, whether either side raised an error, and which requirement is supposed to decide the result. If you skip that last part, you will accumulate mismatches without learning which behavior you actually want to keep. ## How do I avoid false mismatches? Normalize before comparing. Many replacements disagree only in representation: ordering of equivalent results, floating-point formatting, duplicate elimination, tie-breaking, or error wording. Compare canonical forms such as sorted output lists, rounded numeric tolerances, structured error categories, or semantic equality instead of raw bytes. Property-based testing is useful here because it lets you express the comparison once and generate many inputs. [Hypothesis’ introduction](https://hypothesis.readthedocs.io/en/latest/tutorial/introduction.html) shows the basic pattern: generate inputs with strategies, then assert a property over each generated case. In replacement work, that property is often “old(x) and new(x) are equivalent after normalization.” ## How do I decide who is right when they disagree? Use an adjudication ladder: 1. Check the written specification or business rule. 2. Check invariants that must always hold. 3. Minimize the failing input. 4. Add the resolved case as a permanent regression test. Worked example: suppose you replaced a path canonicalization algorithm. - Old output: `/a/c` - New output: `/a//c` - Raw comparison says mismatch. - Normalization collapses repeated separators. - Both become `/a/c`, so this was not a product bug. Now consider input `/a/../..`. - Old output: `/` - New output: error `path escapes root` This is a real semantic disagreement. The deciding question is not which implementation is older. The deciding question is what your contract says about escaping root. Once you choose, encode that rule directly as a test and keep the differential test for discovery. ## How do I generate enough cases to trust the replacement? Use two generators together. Property-based tests explore valid structured inputs and are especially good for equivalence, round-trip, and invariant checks; [Hypothesis ghostwriter](https://hypothesis.readthedocs.io/en/latest/reference/integrations.html) even includes an `--equivalent` mode for this style of test. Coverage-guided fuzzing pushes into unusual parser and state-machine paths. [LibFuzzer](https://llvm.org/docs/LibFuzzer.html) documents that it “tracks which areas of the code are reached” and evolves inputs to maximize coverage. Tool prerequisites: Python with `hypothesis` installed for property-based tests; Clang/LLVM for libFuzzer in C or C++ codebases. Decision checklist: - Define canonical comparison rules. - Separate value mismatches from error mismatches. - Write 5 to 20 spec-backed examples first. - Add invariant checks that both versions must satisfy. - Differential-test old vs. new across generated inputs. - Minimize and classify every disagreement. - Promote each resolved disagreement into a regression test. ## What if both implementations share a bug? That is the main limitation. Differential testing only finds differences, so shared bugs pass through unnoticed. Counter that with independent invariants, round-trip properties, third-party test corpora, and sanitizers during fuzzing runs. LibFuzzer is commonly paired with sanitizers specifically to turn hidden memory and undefined-behavior issues into visible failures. ## Which intentional differences need explicit cases? Any changed tie-breaker, stricter validation, performance guardrail, or [error contract](https://www.devobs.io/articles/stable-api-error-contracts/) should get a named regression test. Otherwise the team will keep rediscovering the same mismatch and arguing about whether it is a bug. Next step: write one normalization function, one differential property test, and one mismatch triage table before changing any more algorithm code. Reviewed: 2026-09-05 ## Source references - - -