# Choose a safe Kubernetes autoscaling signal for queue workers > Pick the metric closest to the outcome you must protect, then turn it into a stable per-pod autoscaling target with clear guardrails. Canonical URL: https://www.devobs.io/articles/qa-ge50-choose-a-safe-kubernetes-autoscaling-signal-when-backlog-latency-and-processing-time-disag/ By: Ines Costa Published: 2024-01-07T06:20:59.366Z Updated: 2026-09-06T10:18:15.722Z Section: Architecture Pick the metric closest to the outcome you must protect, then turn it into a stable per-pod autoscaling target with clear guardrails. For queue consumers, that usually means queue age or estimated drain time derived from backlog and effective throughput, not raw CPU. CPU is safest only when work is fairly uniform and compute-bound. When signals disagree, trust the metric nearest the [SLO](https://www.devobs.io/articles/slo-from-critical-user-journeys/), and add guardrails for delayed metrics, skewed job cost, and scale-up lag. ## What are you actually scaling to protect? Start by naming the workload and the failure you care about: request-serving latency, queue delay, burst drain time, or stream lag. Kubernetes HPA is a periodic control loop, not a continuous one, and it computes desired replicas from the ratio of current to target metric values, as described in the [Kubernetes HPA documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). That matters because a metric that looks useful in dashboards can still be unsafe if its average no longer tracks the protected outcome. For a web API, CPU or request rate may track load well enough. For a worker fleet, CPU often does not. A consumer can be blocked on storage, rate-limited by another service, or waiting on acknowledgements while backlog grows. ## Which signal should win when backlog, latency, and processing time disagree? Use this order: 1. Pick the user-visible objective. 2. Choose the metric closest to that objective. 3. Normalize it by pod capacity if possible. If your SLO is queue delay, use queue age or estimated drain time. If your goal is clearing bursts within a fixed window, backlog per pod can work well. Processing time is usually an input into capacity modeling, not the main scaling signal, because it often reflects job mix more than current shortage. Worked example: your queue must stay under 120 seconds of delay. Under realistic load, one pod completes 20 jobs per second. A burst of 24,000 jobs therefore needs 1,200 seconds on one pod. To drain in 120 seconds, you need about 10 effective pods. That gives you a practical starting target such as backlog-per-available-pod near 2,400 jobs, then adjust for startup lag and skew in load tests. The catch is skew. If one poisoned message or a handful of hour-long jobs tie up consumers, backlog count can stay moderate while oldest-message age explodes. In that case, queue age is the safer primary signal, or you should split slow and fast job classes into separate queues. ## Where does each signal fail? CPU fails for I/O-bound or externally throttled workers. Backlog count fails when work items vary wildly in cost. Latency fails when it is only a downstream symptom; adding pods will not fix a saturated database. External metrics fail when they arrive late, flap, or disappear. HPA also has behavior you need to design around. It averages per-pod metrics, applies tolerance, and treats missing metrics specially, according to the [HPA algorithm details in the Kubernetes documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). In the [Kubernetes HPA walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/), replica counts can take some time to settle under changing load. If your source is event-driven, [KEDA’s scaling model](https://keda.sh/docs/latest/concepts/scaling-deployments/) is worth evaluating because it integrates event-derived metrics with Kubernetes autoscaling. ## What guardrails make the choice safe? Keep the control loop simple and separate demand signals from hard safety limits. Set min replicas for cold-start coverage and max replicas from downstream limits, not just cluster size. Add stabilization to avoid thrashing. Decide what happens when the metric pipeline goes missing. If workers can open too many concurrent downstream operations, enforce that in the worker itself; autoscaling cannot replace concurrency control. **Should we combine CPU and backlog in one HPA?** Only if either metric is allowed to request more replicas. HPA computes a desired replica count for each metric and selects the largest, subject to controller limits and behavior. CPU is not a limiter on backlog-driven scale-out. Use `maxReplicas` and worker concurrency limits to protect downstream capacity, as described in the [Kubernetes HPA algorithm documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). **Should we scale on p95 processing time?** Usually no. Use processing time to estimate per-pod throughput and detect skew, not as the main control input. Next step: write down one explicit queue-delay or drain-time objective, measure effective jobs-per-second per pod under realistic concurrency, and load-test whether your chosen metric still predicts pain during bursts, skewed work, and missing metrics. Reviewed: 2026-09-06. ## Source references - - -