How do we tell a container memory limit failure from node memory pressure?

Check the failure boundary in this order: container status, then pod events, then node conditions around the same timestamp. If the container terminated with OOMKilled, you are usually looking at that container exceeding its own memory limit. If the pod shows eviction events and ends Failed while the node is under memory pressure, the kubelet reclaimed resources at the node level. That difference tells you whether to tune the application or reduce shared-node pressure.

What is the quickest way to classify the failure?

Start with kubectl describe pod or the pod YAML and inspect the last terminated container state before you look at cluster-wide metrics. Kubernetes states that memory limits are enforced by the kernel with OOM kills, and that enforcement is reactive rather than immediate, so a container can briefly exceed its limit before being killed. That is the core signal for a container-scoped failure in the Kubernetes resource management documentation.

Then check pod events. The Kubernetes node-pressure eviction documentation says the kubelet proactively terminates pods to reclaim node resources and sets evicted pods to phase Failed. If events mention eviction, that is your stronger signal that the node, not just the process, ran short.

The ordering matters. First classify the container outcome, then correlate it with pod events, then confirm the node condition at that same time boundary. Do not start with a generic dashboard screenshot and work backward.

What points to a container memory limit failure?

Look for this pattern:

• Terminated container reason is OOMKilled
• Pod is restarted by its controller or restart policy
• No eviction event appears on the pod
• Other pods on the node are stable

In that case, the next change is inside the workload boundary: profile heap growth, reduce in-memory buffers, or raise the memory limit if the usage is legitimate. Also revisit the memory request. Kubernetes schedules from requests, so a low request with a higher real footprint can still pack too many pods onto a node even if the immediate symptom was OOMKilled.

What points to node memory pressure?

Look for the opposite pattern:

• Pod events mention eviction
• Pod phase becomes Failed
• Node reports memory pressure or eviction thresholds were crossed
• Several pods on the same node are affected

The node-pressure doc explains that kubelet compares signals such as memory.available against eviction thresholds. That means a healthy process can still be evicted because the node needed memory back. The next change is usually node- or scheduling-focused: raise requests to reflect reality, spread workloads better, reduce overpacking, or add node memory.

What should we change next?

Use this checklist:

1. If reason is OOMKilled, inspect the container limit and app memory profile.
2. If events say evicted, inspect node pressure and competing pods on that node.
3. If requests are far below observed usage, raise requests first to improve placement.
4. If limits sit below normal peaks, raise limits or reduce memory demand.
5. If several right-sized pods are still evicted, add capacity or rebalance workloads.

Worked example: a pod requests 256Mi, limits at 512Mi, and restarts with OOMKilled while the node has no eviction events. That is an application or limit-sizing problem. If the same workload instead ends Failed with eviction events during a node-wide shortage, fix contention and capacity before changing only that pod.

Does a restart by itself prove OOM?

No. A Deployment can replace both an OOM-killed pod and an evicted pod. The useful distinction is the container termination reason versus the eviction evidence.

What if the failed pod is already gone?

Follow the same evidence boundary: retained pod events first, then node events and node conditions, then metrics around that timestamp. Preserve those three data sources in your runbook so the diagnosis survives pod churn.

Next step: add a short incident checklist to your team docs with the exact commands for those three checks, and require engineers to classify the failure before changing requests or limits.

Reviewed: 2026-09-05