Cloudflare traces billing delays to ClickHouse query-plan contention

Cloudflare published an engineering account on May 14 of a ClickHouse bottleneck that slowed its usage-billing pipeline. The investigation traces the regression to a partitioning change that increased the number of data parts, even though individual queries still read roughly the expected amount of data.

Real-time traces exposed waiting, not work

Ordinary I/O and memory metrics did not explain the slowdown. CPU-sampled flame graphs showed heavy time in query planning, but switching ClickHouse’s trace_log analysis to wall-clock traces revealed that more than half of leaf-query duration was waiting on an exclusive mutex protecting the active-parts list. Every planner acquired the lock, copied the full list, and only then filtered it.

Cloudflare changed the planner to use a shared lock for read-only access, then avoided copying the full vector by maintaining a shared snapshot. Those improvements were merged upstream and available from ClickHouse 25.11, according to the post. A later binary-search optimization used the leading namespace partition key to narrow the range; Cloudflare reports that its March 2026 deployment cut query duration by 50%.

Cardinality can tax planning before execution

The failure is a reminder that partition pruning has a planning cost. A query may touch little data yet still examine metadata for a large part population under concurrency. Teams should graph planning latency against part count and include waiting threads when profiling, rather than relying only on CPU samples.

Partition changes deserve load tests with projected retention and tenant cardinality, not just a small fresh table. Upstreaming generic fixes lowers fork maintenance, but the binary-search optimization here depends on Cloudflare’s query and partition shape. Other deployments should verify their own predicates before expecting the same result.