# How should a system carry timeouts across process boundaries when monotonic clocks cannot be serialized? > Carry an absolute expiry timestamp or a remaining time budget across process boundaries, then convert it once on receipt into a local monotonic deadline. Canonical URL: https://www.devobs.io/articles/qa-ge50-monotonic-deadlines-across-boundaries/ By: Maya Chen Published: 2025-02-19T08:33:39.697Z Updated: 2026-09-06T08:31:04.426Z Section: Architecture Carry an absolute expiry timestamp or a remaining time budget across process boundaries, then convert it once on receipt into a local monotonic deadline. Wall time is shareable but sensitive to clock skew and adjustments; monotonic readings are useful for elapsed time but lack a portable epoch across hosts and reboots. Define the wire contract explicitly rather than serializing a runtime’s internal deadline representation. ## Why not serialize a monotonic deadline? Because a distributed protocol cannot assume a shared clock origin. A system-wide OS clock can be shared by processes in the same clock domain: Linux describes CLOCK_MONOTONIC as system-wide in [clock_gettime(2)](https://man7.org/linux/man-pages/man2/clock_gettime.2.html). That does not make its values portable across hosts or reboots. Separately, Go documents that serialized `Time` values omit the monotonic reading and that the monotonic reading "has no meaning outside the current process" in [Go's time package documentation](https://pkg.go.dev/time). Python's monotonic API says its reference point is undefined and only differences between calls are valid in [PEP 418](https://peps.python.org/pep-0418/). Thus a protocol field like `deadline_monotonic_ns` is a trap unless every participant shares the same clock epoch, which normal distributed systems do not. ## What should the boundary API look like? Prefer one of these two shapes: - `deadline_at`: an absolute UTC timestamp, for example an HTTP header or job field. - `timeout_ms`: a remaining budget, for example for short-lived RPC hops. On receipt, do this once: 1. Read current wall time. 2. Compute `remaining = deadline_at - now_wall` or parse `timeout_ms`. 3. Clamp negative values to zero. 4. Convert to `local_deadline_mono = now_mono + remaining`. 5. From then on, use only monotonic subtraction for checks and child timeouts. Do not repeatedly recompute remaining time from wall clock during execution. Relative timers are intentionally insulated from wall-clock changes on Linux: [clock_gettime(2)](https://man7.org/linux/man-pages/man2/clock_gettime.2.html) notes that changing `CLOCK_REALTIME` affects absolute timers, while timers for a relative interval are unaffected. ## How do you split budgets across retries and downstream calls? Use a single parent budget and spend from it deliberately. Example: an inbound HTTP request arrives with `deadline_at=2026-09-05T20:46:10Z`. Your service computes 1.2 seconds remaining and sets a local monotonic deadline. Decision checklist: - Reserve 150 ms for response marshalling and network writeback. - Allow at most 2 upstream attempts. - Give each attempt `min(400 ms, remaining_after_reserve / attempts_left)`. - Derive the database timeout from the current attempt budget, not the original request budget. That yields consistent behavior: if the first attempt spends 280 ms, the second attempt gets whatever remains after reserve, not a fresh full timeout. This avoids accidental timeout inflation across retries. ## What changes for queues and background jobs? Persist transferable facts, not local clock state. A job record should store either `expires_at` or a service-level maximum age plus enqueue time. When a worker picks up the job, it recalculates remaining time from current wall time, then converts that result to a local monotonic deadline for execution. Queue delay is the critical difference from RPC. A `timeout_ms=30000` value attached at enqueue time is ambiguous if the job waits 25 seconds before pickup. `expires_at` is usually better for async work because it preserves intent across storage and delay. ## Which failure modes matter? Three matter immediately. First, NTP or manual clock steps can make shared wall-clock deadlines appear to move; that is why you convert once and then run on monotonic time. Second, suspend behavior differs by clock. Go warns that on some systems the monotonic clock stops during sleep in [Go's time package documentation](https://pkg.go.dev/time), and Linux distinguishes `CLOCK_MONOTONIC` from suspend-aware `CLOCK_BOOTTIME` in [clock_gettime(2)](https://man7.org/linux/man-pages/man2/clock_gettime.2.html). Third, precision is limited: OpenTelemetry defines timestamps and durations as epoch-based timestamps and elapsed durations with language-specific representation in the [OpenTelemetry trace API](https://opentelemetry.io/docs/specs/otel/trace/api/). Log both the wall-clock event time and the computed remaining budget, but do not imply impossible nanosecond accuracy end to end. ## Follow-up Q&A **Should I propagate both deadline and remaining budget?** Usually no. Pick one canonical wire representation per boundary. For synchronous RPC, `deadline_at` is easier to inspect across hops. **Should workers drop expired jobs before starting?** Yes, if the enforced boundary is job start. Check expiry immediately after dequeue, then transition the job to expired or skipped before doing side effects. Next step: define one canonical deadline field for your HTTP and job protocols, then add a tiny library that converts it to a local monotonic deadline exactly once at process entry. Reviewed: 2026-09-05 ## Source references - - - -