Quarantine flaky tests without teaching CI to ignore failures

A flaky-test quarantine should remove a known nondeterministic result from the release gate while preserving the test and its visible failure history, plus a time-bound obligation to repair it. Quarantine is a workflow state with an owner and expiry, not a skip annotation that quietly becomes permanent.

Require evidence before entry

A flaky test is one that intermittently passes and fails without an intentional code change, often due to uncontrolled state, timing, ordering, or environment. The pytest flaky-tests guide describes intermittent or sporadic failures with non-deterministic behavior and points to common causes including uncontrolled state, ordering, overly strict assertions, and thread-safety problems. Google’s post on flaky tests at Google defines a flaky result as one that shows both passing and failing outcomes with the same code.

Capture the failing seed, environment, runner, dependency versions, timing, logs, and several reruns before classifying a test as flaky. The pytest flaky-tests guide discusses rerun and detection tools, but a passing rerun does not make the original failure irrelevant.

The quarantine record should contain test identifier, owner, issue link, entry evidence, suspected cause, date, deadline, affected platforms, and exit criteria. Require review for tests covering security, migrations, money movement, or other critical behavior; quarantine may be the wrong response when the failure itself signals unacceptable uncertainty.

Keep the signal visible but separate

Run quarantined tests in every applicable CI cycle or on a clearly defined frequent schedule. Publish their failures in the normal test report and trend them separately. They should not block the release while the quarantine is valid, but missing execution should. Make the release summary say how many quarantined tests ran, failed, expired, or recovered.

Google describes monitoring flakiness, quarantining overly flaky tests to remove them from the critical path, and filing bugs for developers to reduce flakiness. That distinction matters: quarantine is not the same as ignoring the test. The organization still pays investigation time, compute, and lost confidence even when CI turns green, as both the pytest flaky-tests guide and Google’s flaky-tests post explain.

Avoid unlimited automatic reruns on the gated suite. They increase feedback time and can normalize real regressions as intermittent noise. If reruns are used to diagnose, report the first failure and attempt count. Track flaky failure rate, rerun compute, quarantine age, and recurrence after repair. Do not turn a low global flake percentage into a target that excuses specific old tests.

Give quarantine an expiry path

At the deadline, either repair and verify, obtain a reviewed extension with new evidence, or delete the test after demonstrating that the behavior is covered elsewhere or no longer exists. The pytest flaky-tests guide explicitly includes deleting or rewriting the test when other coverage exists or a lower-level rewrite would make the source of flakiness clearer. An expired quarantine should fail a dedicated policy check even if the test happens to pass that day. This keeps ownership visible without making a random test failure block unrelated delivery.

Exit requires more than one green run. Reproduce the failure before the fix when possible, add a focused assertion or deterministic harness, then run enough repeated and varied executions to exercise the former failure mode. Remove the quarantine metadata in the same change and monitor recurrence.

Install the contract before the label

Create a machine-readable quarantine manifest and one CI job that always executes it. Define entry evidence, maximum initial duration, extension authority, critical-test exclusions, and exit verification. Migrate existing skipped flaky tests into records with owners; if no one owns one, treat that as an immediate triage issue.

For the next flaky failure, preserve the first artifact, open the repair issue, and measure how much time reruns consume. That single case will test whether the process protects delivery while retaining enough signal to fix the underlying defect.

Preserve release-risk context

A quarantined test can overlap changed code. When a small pull request or dependency stack touches its owning module, surface the quarantine prominently and require a targeted risk decision. Run the test against the change even though it does not gate. If failures become more frequent, deterministic, or appear on new platforms, automatically reopen triage; the original flake label must not hide a new regression. Preserve artifacts from both the first failure and every diagnostic rerun.

Reviewed September 2026.