K8S Tips and Scripts #1

I’ve often come across situations where pods won’t die properly. Either the underlying node is in a strange state, temporary networking issues, whatever. If a node has gone south, pods will often stick in an “Evicted” state for example.

Delete All Misbehaving Pods

There can also be events which will leave hundreds of pods in strange state. Very often, all that is needed is to delete the pods. Doing it manually is toil.

Here is a simple one liner which will loop through the namespaces which have pods in strange states and delete those pods. Either copy/paste the output or feed it directly into bash with a pipe as I have done here..

$ kubectl get po -A | awk '/Evict|Crash|Error/ {ns[$1]=ns[$1] " " $2} END {for (i in ns) printf "kubectl delete po %s -n %s\n", ns[i],i}' | bash
pod "traefik-5c6db74dbf-949d2" deleted
pod "traefik-5c6db74dbf-qbcmm" deleted
pod "traefik-5c6db74dbf-rjhcf" deleted
pod "traefik-5c6db74dbf-wjcbj" deleted

The output looks something like this, with one line per namespace:

kubectl delete po traefik-5c6db74dbf-949d2 traefik2-5c6db74dbf-qbcmm .. -n kube-system

I like scripts that effectively compile to another script which actually
gets executed. This way I can examine it before executing it. I use that
type of design philosophy often as it gives me confidence that
it will do what I want (and nothing more).