Sample traces for failures you did not predict

Use head sampling to preserve a stable, representative view of normal traffic and tail sampling to retain diagnostically valuable traces after their outcome is known. Neither method alone is enough when rare errors and latency outliers matter but collecting every span is unaffordable.

Know when the decision is made

Head sampling decides near trace creation, before the system knows whether the request will fail three services later. It is cheap, distributes naturally, and gives predictable volume. Its weakness is probability: a one-in-ten-thousand failure can disappear in a one-percent sample.

Tail sampling waits until enough spans arrive at a collector to evaluate the trace. The OpenTelemetry sampling documentation describes head and tail approaches and their placement in the telemetry pipeline. Tail policies can retain errors, long duration, specific routes, or unusual attributes, but require buffering and coherent routing of spans.

Trace identity and sampling intent cross process boundaries through context propagation. The W3C Trace Context recommendation defines traceparent and tracestate fields, including the sampled flag. Preserve that context across HTTP, queues, and jobs; a broken propagation path fragments the evidence an incident responder needs.

Build a portfolio of policies

Start with a small probabilistic head sample of all eligible traffic. This provides baselines for healthy latency, service topology, and comparison during an incident. Stratify where traffic differs sharply: a low-volume payment route should not compete with a high-volume health endpoint under one blunt rate.

At the tail, retain all traces with server errors, selected domain failures, or latency beyond route-specific thresholds. Preserve traces with rare operational attributes such as a failover region, retry exhaustion, circuit opening, or an experimental release. Add a small random tail sample of everything else so policies do not define your entire view of reality.

Avoid sampling based on uncontrolled high-cardinality values such as user ID. That can leak sensitive identifiers into policy configuration and produce unpredictable volume. Normalize routes and explicitly allow useful attributes.

Plan for collector failure

Tail sampling consumes memory until a trace is complete or times out. Late spans, long jobs, and fan-out increase the holding period. Send all spans for a trace to the same decision point, estimate concurrent traces rather than only spans per second, and set a maximum wait.

Overload behavior must be explicit. Dropping the newest spans can erase an active incident; keeping only errors can destroy the healthy comparison group. Expose received spans, incomplete traces, decision latency, dropped traces by reason, queue saturation, and exporter failures.

Use a two-tier pipeline when needed: local collectors batch and forward; a consistent trace-ID routing layer sends complete traces to tail-sampling collectors. Test redistribution during autoscaling because moving a trace mid-flight can split it.

Recognize sampling bias

An “all errors” sample does not reveal the error rate unless you also know the inclusion probability and total request count. Metrics remain the source for rates and service-level indicators. Traces explain individual paths.

Tail latency thresholds must vary by operation. A 900-millisecond report may be healthy while a 200-millisecond autocomplete is not. Derive thresholds from service objectives, then revisit them as behavior changes.

Record the active sampling policy version on telemetry or in deployment metadata. During an incident, responders need to know whether absence means no event or an exclusion rule.

Test the incident you have not named

Replay traffic with one rare error, one slow dependency, one large fan-out, and normal requests. Verify that the rare traces survive, healthy context remains, and collector saturation is visible. Then disable one collector and repeat.

Set a concrete budget in traces or bytes per second. Allocate it among representative traffic, error retention, latency outliers, and experimental rules. The next step is to inspect last month’s incidents and ask whether the current policy would have retained the first failing trace and a comparable successful one.