How do I prove a regression test fails for the bug it is meant to prevent?

A regression test is most convincing when the same behavioral check fails on an affected revision and passes on the fixing revision. The key is not just “red before, green after,” but “red for the same reason as the bug.” In practice, run the test against the parent or known-bad commit in an isolated checkout, classify any failure you see, then rerun the identical behavioral assertion on the fix. If the API changed, adapt the test harness, not the claim being tested.

What counts as proof?

A practical proof bar is this: the test should demonstrate the reported behavior on a bad revision and its absence on the fixed revision. If the old revision crashes during setup, will not compile, or fails an unrelated assertion, you have not proved the regression test yet.

This is where good assertion output matters. pytest assertion introspection shows the values of failed comparisons, and pytest.raises is useful for checking the exception path and matching the bug-specific symptom. Use those features to show that the failure matches the original symptom, not just any failure.

How should you run before-and-after checks?

Use separate working directories so you do not contaminate one run with another. Git worktree exists to “manage multiple working trees,” and its add command can create another checkout of the same repository while “sharing everything except per-worktree files.” That is exactly what you want for bug reproduction on parent and fix revisions.

A practical flow:

1. Identify the fix commit and its parent, or another known-affected commit.
2. Create two worktrees: bug-before at the bad revision and bug-after at the fix.
3. Install the dependencies appropriate to each revision.
4. Run the same reproducer in both places.
5. Record the result as one of three categories:
   • expected behavioral failure
   • infrastructure failure
   • unrelated behavioral failure

Only the first category proves the regression test.

What if the patch changed the public interface?

Keep the behavioral expectation constant and adapt only the harness needed to invoke it. For example, if parse_user(data) became parse_user(data, *, strict=True), your regression claim is still “malformed input should raise ValueError” or “duplicate IDs should be rejected,” not “call this exact function signature.”

A minimal pytest example:

# Prerequisite: pytest installed in both worktrees.
import pytest

# before revision harness
from app import parse_user

def test_duplicate_id_is_rejected():
    with pytest.raises(ValueError, match="duplicate id"):
        parse_user({"id": "42"}, {"id": "42"})

If the fixed revision changed the signature, adapt only the invocation harness and preserve the same behavioral expectation. Keep the same exception type and, when the original symptom depends on it, a stable message pattern. pytest.raises is useful here because it checks both the exception path and, with match, a bug-specific symptom when that symptom is part of what you need to prove.

What if the old revision will not build or the bug is intermittent?

If the old revision will not build, say so explicitly and downgrade the proof level. You may still have a strong regression test, but not a demonstrated before/after proof on that exact parent. Try a nearby affected tag, containerized historical dependencies, or a narrower unit-level reproducer that avoids the broken build path.

If the bug is intermittent, prove probability changed, not certainty. Freeze time, seed randomness, isolate I/O, and mock unstable dependencies. Your goal is to turn a flaky reproducer into a deterministic one before you call it a regression test.

Decision checklist

• Did the same behavioral assertion run before and after?
• Did the bad revision fail for the bug’s reason?
• Did the fixed revision pass without weakening the assertion?
• If the interface changed, did only the harness change?
• If proof was impossible, did you document exactly why?

Follow-up: What if the test only fails after the fix because the assertion got stricter?

That is not a valid regression proof. Re-run the exact same behavioral expectation on the bad revision first.

Follow-up: Can I use a different test at the bad revision and the fixed revision?

Usually no. You can adapt setup code for interface drift, but the observed behavior and pass/fail rule should remain the same.

Next step: pick the fix commit, create two worktrees, and write down the exact failure category you see on the parent before you call the test a regression proof.

Reviewed: 2026-09-06.