Why is averaging server p99 latency misleading for a fleet?

Averaging p99 latency across servers is misleading because a percentile is already a summary of a distribution, not a raw measurement you can safely average. Once each instance publishes only its own p99, you have lost the information needed to weight by request volume and reconstruct the fleet-wide tail. For a service-wide latency decision, aggregate mergeable histogram data across instances and compute the percentile from the combined distribution instead, as recommended by Prometheus histogram guidance and supported by the OpenTelemetry metrics data model.

Why can’t you average p99s across pods?

Because percentiles are not linear. A fleet p99 is the latency below which 99% of all requests across the fleet fall. An instance p99 is the same idea, but only for that instance’s requests. If instances handle different request counts, averaging their p99 values gives equal influence to unequal populations.

Prometheus histogram guidance says this plainly: you “cannot aggregate quantiles” for a service backed by multiple workers. That is the core mistake. A p99 from 100 requests and a p99 from 1,000,000 requests are not peers you can average meaningfully.

What does the aggregation mistake look like with real numbers?

Take two pods over the same minute:

• Pod A handles 10,000 requests. Its p99 is 120 ms.
• Pod B handles 100 requests. Its p99 is 2,000 ms.

If you average the two p99s, you get 1,060 ms.

The average gives equal weight to pods that handled very different request counts. These two p99 values do not determine the fleet p99: you need the combined request distribution to locate its 99th-percentile boundary.

You can also fail in the opposite direction: two pods can report the same p99 while having very different tail shapes above or just below that point. Averaging identical p99s still tells you nothing about the combined distribution. The problem is not only weighting; it is information loss.

What should you aggregate instead?

Aggregate histograms. Prometheus histogram guidance distinguishes summaries from histograms and recommends histogram-style representations because the distribution is represented in a way that “can be aggregated with each other.” That is exactly what you need for fleet latency.

The OpenTelemetry metrics data model supports this at the transport and collection layer by defining OTLP streams with an “intrinsic decomposable aggregate function” that makes it “semantically well-defined to merge data points across both temporal and spatial attributes.” In practice, that makes histogram data suitable for cross-instance merging before downstream percentile estimation.

For practical engineering decisions, choose bucket boundaries around the threshold you care about. If your question is “Are 99% of requests under 300 ms?”, add tighter buckets near 300 ms than at multi-second ranges. Percentiles derived from histograms are approximate, and the approximation quality depends on bucket layout near the tail boundary you will alert on or use for capacity decisions.

What if histogram boundaries differ?

In practice, use one canonical bucket boundary scheme for a given latency metric across the fleet. Mixed boundaries complicate aggregation and can reduce accuracy. Standardizing boundaries per latency metric is the simplest operational rule.

Can old summary data recover the fleet p99?

No. If you only stored per-instance summary quantiles, you cannot reconstruct the true fleet p99 later. Prometheus histogram guidance explicitly warns that you cannot recalculate other quantiles or aggregate those quantiles into a service-wide percentile.

Next step: pick one fleet latency metric, switch it to histogram instrumentation end to end, standardize bucket boundaries around your SLO threshold, and compute p99 from the merged fleet distribution rather than from averaged instance p99s.

Reviewed: 2026-09-06