CI needs more instrumentation about kepler HOT 10 CLOSED

https://github.com/sustainable-computing-io/kepler/actions/runs/3073381944/jobs/4965474260#step:8:233
I see latest CI passed, @rootfs is this issue able to reproduce easily?

from kepler.

let's keep an eye on it and re-open this issue if it happens again.

from kepler.

Maybe we need a larger timeout?
Let's see if it happen again

from kepler.

#231 (comment)

from kepler.

need further investigate with https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging

from kepler.

Thanks @SamYuan1990!

What we need is the artifacts https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts

So we will dump the logs and description of the pods to files in the artifact folder
Something like:

kubectl describe pod -n $1 $2 > $artifacts/describe/pod/$2
kubectl logs -n $1 $2 > $artifacts/logs/pod/$2

But, to save space we just dump the info when something fail

from kepler.

Thanks @SamYuan1990!

What we need is the artifacts https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts

So we will dump the logs and description of the pods to files in the artifact folder Something like:

kubectl describe pod -n $1 $2 > $artifacts/describe/pod/$2
kubectl logs -n $1 $2 > $artifacts/logs/pod/$2

But, to save space we just dump the info when something fail

any suggestion for fail handle?
I suppose our points is

when fail happnes.
kubectl describe pod -n $1 $2 > $artifacts/describe/pod/$2
kubectl logs -n $1 $2 > $artifacts/logs/pod/$2

from kepler.

@SamYuan1990 something like:

function logdump {
   kubectl get po -A -o custom-columns=namespace:.metadata.namespace,name:.metadata.name | tail +2 | while read -r line ; do 
     line=($line)
     namespace=${line[0]}
     podname=${line[1]}
     
     mkdir -p $artifacts/describe/pod/$namespace
     mkdir -p$artifacts/logs/pod/$namespace
     
     kubectl describe pod -n $namespace $podname > $artifacts/describe/pod/$namespace/$podname
     kubectl logs -n $namespace $podname > $artifacts/logs/pod/$namespace/$podname
   done
   
}

trap 'logdump' ERR

from kepler.

@rootfs ,is this issue able to be reproduced after we fixed panic issue?

from kepler.

@SamYuan1990 we are not seeing it recently.

from kepler.