How do I make a performance regression test useful on shared CI machines?

Shared CI can support a useful performance regression test, but only if you treat it as a coarse merge gate rather than a lab-grade benchmark. Measure the natural variance first, run a fixed representative workload repeatedly, compare results against a baseline produced in the same environment, and fail only on regressions clearly larger than that noise band. Keep deeper benchmark diagnostics separate from the gate.

What should the CI gate actually decide?

The gate should answer one narrow question: did this change make a representative operation slower by more than the normal run-to-run variance of this CI environment? That is different from asking for precise throughput numbers or root-cause analysis.

On shared runners, scheduler noise, CPU frequency changes, and contention are real. The pyperf system tuning guide explicitly says that isolating cores has a “significant impact on the stability of benchmarks,” and that CPU pinning makes benchmarks “more stable.” The Google Benchmark user guide likewise documents warmup, repetitions, standard deviation reporting, and random interleaving to reduce the impact of changing system state.

That means your gate should be conservative by design: only catch obvious regressions, not every small movement.

How do I choose a threshold that survives noisy CI?

Start by measuring variance before you write a failure rule. Run the same benchmark on the same branch in CI at least 20 to 30 times over different builds. Record the mean and spread. If your benchmark framework can report repeated-run statistics, use them; Google Benchmark reports “mean and standard deviation” when repetitions are greater than 1.

A practical rule is to set the gate above the observed noise floor. For example:

• baseline endpoint test: 120 ms mean
• normal CI variation: plus or minus 6%
• gate threshold: fail only above 12% regression

That threshold is intentionally wider than the routine variance. It will miss tiny regressions, but it will stop noisy false failures from training the team to ignore the signal.

Also fix the workload carefully: same input sizes, same dataset shape, same concurrency, same warmup. If the benchmark needs caches or JIT warmup, use a warmup period; Google Benchmark supports a minimum warmup time and discards those early results.

How should I compare results fairly?

Compare artifacts under the same benchmark conditions: build the candidate artifact, run the benchmark job, and compare its output with a stored baseline produced by the same benchmark definition on the same runner class. Do not compare a PR run on one machine type with a six-month-old laptop result.

A simple CI pattern is:

1. Run one small benchmark suite on every PR.
2. Use repetitions, such as 5 to 10 runs per case.
3. Store JSON output from the benchmark tool.
4. Compare the candidate mean against the baseline mean.
5. Fail only if the change exceeds your justified threshold.
6. Publish all raw runs for human review.

If you use Google Benchmark, --benchmark_repetitions, --benchmark_min_warmup_time, --benchmark_out, and --benchmark_enable_random_interleaving are directly relevant here.

What belongs outside the merge gate?

Anything diagnostic. Hardware counters, many scenario permutations, long-running suites, and exploratory profiling are better as scheduled jobs or dedicated benchmark environments. Shared CI is good for a narrow smoke alarm, not for proving a 2% win.

Worked decision checklist?

Use the gate if all of these are true:

• the workload matches a real hot path
• inputs are fixed and versioned
• repeated runs show an understandable variance band
• your threshold is larger than that variance
• the benchmark output is stored for review
• an intentional slowdown can be approved by updating the baseline in the same change

If those are not true, keep the benchmark as a report, not a blocker.

When is dedicated hardware necessary?

When you need to detect small regressions, explain causes, or compare branches with high confidence. If the regression threshold you need is close to the CI noise floor, shared runners are the wrong tool.

How should intentional slowdowns be reviewed?

Treat them as explicit tradeoffs. Include the code and benchmark result in the same change, and update the baseline only after reviewers accept the slower path for a reason such as correctness or safety. That keeps the benchmark honest instead of turning every slowdown into a silent reset.

Next step: pick one production-relevant operation, measure its variance on your current CI for a week, and set the first threshold from those numbers rather than intuition.

Reviewed: 2026-09-05