Builder pods not removed after deploy about builder HOT 11 OPEN

From @felixbuenemann on February 20, 2017 21:47

This behavior can easily inspected with:

kubectl get --namespace deis --show-all pods | grep build-

The number of completed pods will increase by one for each build.

from builder.

From @bacongobbler on February 21, 2017 1:15

related: deis/builder#57

This seems like in recent versions of k8s, they stopped cleaning up pods in the "success" state. Probably some research needs to be done on how to turn this functionality back on.

from builder.

From @felixbuenemann on February 21, 2017 9:21

I'm running K8s 1.4.x if that matters.

Regarding #57 suggestion for Jobs – neither Jobs nor Pods are removed automatically.

From the K8s Job docs:

When a Job completes, no more Pods are created, but the Pods are not deleted either. Since they are terminated, they don’t show up with kubectl get pods, but they will show up with kubectl get pods -a. Keeping them around allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output. The job object also remains after it is completed so that you can view its status. It is up to the user to delete old jobs after noting their status. Delete the job with kubectl (e.g. kubectl delete jobs/pi or kubectl delete -f ./job.yaml). When you delete the job using kubectl, all the pods it created are deleted too.

from builder.

From @felixbuenemann on February 21, 2017 10:34

Interestingly the docs on Pod Lifecycle say:

In general, Pods do not disappear until someone destroys them. This might be a human or a controller. The only exception to this rule is that Pods with aphase of Succeeded or Failed for more than some duration (determined by the master) will expire and be automatically destroyed.

This seems to be in contrast to what I'm actually seeing…

from builder.

From @felixbuenemann on February 21, 2017 11:18

I have opened kubernetes/kubernetes#41787 for clarification of the above statement from the docs.

from builder.

From @felixbuenemann on February 27, 2017 22:27

I just got feedback to the kubernetes issue, it looks like by default completed or failed pods are garbage collected if there are more than 12,500 pods. Obviously that is not very helpful in this case, so an automatic cleanup by the builder should be implemented.

from builder.

From @felixbuenemann on March 6, 2017 11:8

Quoting here from the kube-controller-manager help on the --terminated-pod-gc-threshold <n> option:

Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled. (default 12500)

from builder.

From @kwent on March 20, 2017 17:34

Any progress on this ? Sounds like a waste of resources and space for everyone.

from builder.

From @pfeodrippe on April 6, 2017 14:46

Same here, it may be linked to a issue I've opened last week.

$ kubectl get --namespace deis --show-all pods | grep build-
slugbuild-teslabit-web-production-d2fcd4c0-7e507178   0/1       Completed   0          1d

from builder.

From @pfeodrippe on April 12, 2017 14:16

I'm using this tiny git pre-push hook for deletion https://gist.github.com/pfeodrippe/116c8b570ee2ffcdce8aa15bbae5a22b.

It deletes the last slugbuild created for the app when you git push

from builder.

From @davidlmorton on July 25, 2017 2:15

+1 This bit me after a couple of weeks of deploying applications to my deis cluster.

from builder.