# Should applications block, buffer or drop telemetry when the backend is unavailable? > Buffer diagnostic telemetry within fixed limits, then drop it during prolonged outages; give must-keep business records a separate durable path. Canonical URL: https://www.devobs.io/articles/qa-telemetry-backend-outage-loss-policy/ By: Lena Fischer Published: 2024-06-06T13:57:36.428Z Updated: 2026-09-06T10:18:15.722Z Section: Architecture Applications should almost never block user or business traffic just because tracing, metrics, or log export is down. For diagnostic telemetry, the right default is bounded buffering with eventual drop: absorb short outages, retry for a limited time, and then shed telemetry rather than expand latency or crash the app. OpenTelemetry’s error-handling guidance explicitly prefers losing telemetry over significantly changing application behavior, and says SDKs must not fail the application later at runtime because exporters cannot reach their endpoint ([OpenTelemetry error handling](https://opentelemetry.io/docs/specs/otel/error-handling/)). ## When should telemetry never block requests? If the signal exists to diagnose the system rather than to represent a business fact, keep it off the critical path. That includes request traces, runtime metrics, and most operational logs. OpenTelemetry’s guidance is clear: telemetry libraries are not essential to application business logic, and users would generally prefer data loss over application behavior changes ([OpenTelemetry error handling](https://opentelemetry.io/docs/specs/otel/error-handling/)). That means no synchronous “must export before response” contract for normal observability. A slow or unreachable backend should not turn into user-facing latency, stuck workers, or failed writes. ## What should happen instead of blocking? Buffer briefly, with hard limits. The collector resiliency guidance recommends a sending queue plus retries for network exporters, but also explains the two important loss boundaries: data is dropped when the queue fills, and old data is dropped when retry time is exceeded ([OpenTelemetry Collector resiliency](https://opentelemetry.io/docs/collector/resiliency/)). That is the contract most teams actually want: - in-memory queue for short backend blips - exponential-backoff retries for recovery - fixed queue size so memory use stays bounded - explicit dropping once the outage exceeds your telemetry failure budget If collector crashes are part of the risk model, add persistent queue storage such as the collector’s write-ahead-log style file storage. If the hop is truly critical across systems, use a real [message queue](https://www.devobs.io/articles/queues-versus-streams-by-recovery-model/) between collectors instead of pretending observability export is durable by default ([OpenTelemetry Collector resiliency](https://opentelemetry.io/docs/collector/resiliency/)). ## When is dropping the wrong answer? Dropping is wrong when the record itself is the business obligation. Examples: audit evidence required by policy, financial events, security events that must enter an investigation pipeline, or workflow messages that drive downstream state. Those are not “just telemetry.” Treat them as business records with their own durable transport, retention, replay, and verification model. You can still mirror them into observability tooling, but the application should not rely on an observability exporter as the system of record. ## How do you choose a practical policy? Use this checklist: 1. Ask whether the signal is diagnostic or business-critical. 2. For diagnostic signals, prohibit request-path blocking. 3. Set a small bounded queue and retry window sized for common outages, not worst-case disasters. 4. Decide where loss is acceptable: SDK, sidecar, node agent, or gateway collector. 5. Emit self-observability for queue depth, enqueue failures, and send failures. 6. For must-keep records, move them to durable messaging or storage first. Worked example: an API service exports traces through an OpenTelemetry Collector. The collector keeps a 5,000-batch queue and retries for 10 minutes. A 90-second backend restart causes no app impact and most traces drain later. A 30-minute outage fills the queue; newer traces are dropped. That is acceptable for debugging. The same service’s compliance audit events go to Kafka first, then are copied to the observability backend asynchronously. ## What should you monitor during an outage? Watch queue size versus capacity, enqueue failures, and send failures. If those rise during backend trouble, your bounded-loss design is working as intended; if application latency rises instead, telemetry is still too coupled to the request path. The OpenTelemetry guidance specifically calls out monitoring queue metrics to understand whether capacity matches workload ([OpenTelemetry Collector resiliency](https://opentelemetry.io/docs/collector/resiliency/)). **Follow-up Q:** Should error logs block if they are the only clue during an outage? **A:** No. Give them reserved buffer capacity or a separate path, but keep emission bounded. **Follow-up Q:** Should I use disk buffering everywhere? **A:** No. Use persistent storage on critical collectors where restart loss matters; otherwise keep the pipeline simpler. Next step: write down one explicit failure contract per signal type—block, bounded buffer, or durable path—and test it by disabling your telemetry backend for 15 minutes. Reviewed: 2026-09-05 ## Source references - -