How do we find what will break before upgrading a small Kubernetes cluster?

Upgrades break small Kubernetes clusters when something outside the core control plane still depends on an API version, admission behavior, or component version the target release no longer supports. The reliable approach is to build an inventory against the exact target minor version: enumerate live API usage, installed controllers and add-ons, webhook rules, and client/component versions, then rehearse one minor step at a time within the documented version-skew boundaries from Kubernetes’ own policy and deprecation guide.

What should we inventory before touching the control plane?

Start with the target version, not a generic checklist. In the Kubernetes Deprecated API Migration Guide, removals are listed by release, and some include behavior changes, not just a new apiVersion. For example, Kubernetes 1.25 stopped serving batch/v1beta1 CronJob and policy/v1beta1 PodDisruptionBudget, and the PDB selector behavior changed in policy/v1.

For a small cluster, inventory these five buckets:

• Live resources: objects already stored in the cluster that still use removed APIs.
• Manifests and Helm charts: what your Git repo would reapply after the upgrade.
• Controllers and add-ons: ingress controller, cert-manager, metrics stack, CSI driver, CNI, autoscalers, operators.
• Admission webhooks: anything registered in ValidatingWebhookConfiguration or MutatingWebhookConfiguration.
• Clients and node components: kubectl, kubelets, kube-proxy, and any automation using client libraries.

Manifest scanning alone is not enough. It misses old objects still persisted in the cluster, and it does not tell you whether an admission webhook or operator can handle the new object versions and fields.

Where is the hard upgrade boundary?

The hard boundary is Kubernetes version skew. The Kubernetes Version Skew Policy says kubelet must not be newer than kube-apiserver, kubectl is supported within one minor version of the API server, and controller-manager and scheduler must not be newer than the API server. It also gives the supported component upgrade order.

That means your enforced sequence is:

1. Patch current minor to latest patch.
2. Move one minor version at a time.
3. Upgrade the API server first.
4. Then controller-manager and scheduler.
5. Then kubelets and kube-proxy.
6. Upgrade kubectl and automation clients so they stay within supported skew.

For admission webhooks, the policy is explicit: before upgrading the API server, registered webhooks must be able to handle the data the new API server will send. That is a precondition, not a nice-to-have smoke test.

How do we do a bounded rehearsal?

Use a short worksheet for the exact source and target, for example 1.24 to 1.25:

• Check the deprecation guide for 1.25 removals.
• Search live objects and manifests for batch/v1beta1, policy/v1beta1, autoscaling/v2beta1, discovery.k8s.io/v1beta1, and events.k8s.io/v1beta1.
• Confirm each add-on officially supports 1.25.
• Review webhook configurations and verify the backing service handles equivalent or new resource versions.
• Rehearse in a disposable cluster or snapshot-restored environment with the same add-ons.
• During rehearsal, test: deploy, scale, ingress, certificate issuance, metrics/HPA, backups, and one restore.

A useful decision rule: if a component is cluster-critical and you cannot verify target-version support, block the upgrade.

What recovery is still available after the control plane changes?

Assume recovery options shrink after the API server upgrade. If the new API server stops serving an old version, reapplying old manifests will not fix the issue. Your recovery plan should therefore be prepared before the control plane change: etcd backup or provider snapshot, exported manifests, add-on versions pinned, and a tested restore path.

Two follow-up questions:

Do we need to scan client libraries too? Yes. Deprecated and removed APIs break automation just as surely as manifests. Anything creating or updating cluster objects must be checked against the target release.

Can we skip rehearsal for a tiny cluster? Only if the workloads are disposable. Small clusters still depend on webhooks, add-ons, and old manifests, and those are exactly where surprise failures show up.

Your next step is to create a target-version worksheet for the next single minor upgrade and refuse to schedule the maintenance window until every cluster-critical add-on and webhook on that sheet has an owner and a compatibility answer.

Reviewed: 2026-09-05.