A Kubernetes rollout passed in two minutes, then pods crashed at minute five
A delivery tracking API release showed as successful in Kubernetes, but its new pods began failing five minutes later. The team explains why its checks missed the problem.
Source: Dev.to
In May, release 7.3 of a delivery tracking API went out at 11 a.m. The pipeline relied on kubectl rollout status, which reported success after two minutes, so the deploy was marked green. By the seventh minute, all twelve new pods had restarted at least once, and no old pods were left to take traffic.
The release added a background task that sent delivery events to a reporting database, first five minutes after startup and every five minutes after that. Its credentials existed in staging but not production. The Node service's first push failed with an unhandled promise rejection, which ends the process by default since Node 15. Each pod passed its readiness probe, ran for five minutes, then crashed, and soon all were in CrashLoopBackOff.
Pods counted as available as soon as they became ready, because minReadySeconds was never set and defaulted to zero. The controller scaled down the old pods to match, so the replacement finished long before the failures began. The error-rate alert fired at eight minutes, and a rollback took about ninety seconds. The fixes run the task once at startup before readiness, set minReadySeconds to sixty, and keep the pipeline watching restarts and errors for fifteen minutes with automatic rollback. The team's takeaway is that readiness only showed the pod could answer at that moment, not that the release would last.
