Why Kubernetes Rolling Updates Still Drop Requests, and How to Stop It

Kubernetes rolling updates won't take a service down, but they aren't zero downtime by default. A Dev.to guide explains how to fix the gaps at the start and end of a pod's life.

Rolling updates replace pods in steps governed by maxSurge and maxUnavailable, both defaulting to 25%. On a four-replica Deployment, Kubernetes can remove an old pod before its replacement is ready, cutting capacity mid deploy. The author recommends maxUnavailable: 0 with a small maxSurge, plus minReadySeconds to catch pods that crash just after becoming Ready.

At startup, a pod can be Ready in Kubernetes before an external load balancer has registered it as healthy. Readiness gates close that gap: the pod only counts as Ready once the load balancer controller confirms the target is healthy. The AWS Load Balancer Controller can inject the gate for labeled namespaces, and GKE's container-native load balancing adds its own.

At shutdown, SIGTERM is sent while endpoint removal is still propagating, so requests may reach an app that has stopped listening. This is a common source of 502s. A preStop sleep delays shutdown so routing can catch up. The grace period must exceed the delay plus the app's drain time, and the app must handle SIGTERM properly.

PodDisruptionBudgets protect against node drains, upgrades and autoscaler scale-downs, but not Deployment rollouts. Teams should also run kubectl rollout status with a timeout in CI. A stalled rollout reports ProgressDeadlineExceeded after 600 seconds by default and does not roll back on its own, so the pipeline should run an undo.

Source: Dev.to


Posted

in

by