exec probes such as readiness and liveness probes time out in v1.19.3 about bottlerocket HOT 8 CLOSED

Adjusted the title to reflect that the impact isn't Redis-specific in any way.

from bottlerocket.

With the release of Bottlerocket v1.19.4 the containerd issue should be addressed!

from bottlerocket.

Yeap, v1.19.4 has this issue fixed - all Redis Pods are starting normally using Karpenter's EC2NodeClass:

spec:
    amiFamily: Bottlerocket
    amiSelectorTerms:
    - name: bottlerocket-aws-k8s-1.28-x86_64-v1.19.4*
      owner: "651937483462"
    - name: bottlerocket-aws-k8s-1.28-aarch64-v1.19.4*
      owner: "651937483462"

from bottlerocket.

In case anyone is wondering what the exec probes are doing, it simply invoke redis-cli to issue a localhost redis PING command:

$ cat /health/redis_readiness.sh
response=$(
  redis-cli \
    -a "${AUTH}" --no-auth-warning \
    -h localhost \
    -p 6379 \
    ping
)
if [ "$response" != "PONG" ] ; then
  echo "$response"
  exit 1
fi
echo "response=$response"

from bottlerocket.

Thanks for the report! This seems similar #3866 reported yesterday. Could you please confirm if this is occurring consistently or intermittently?

from bottlerocket.

Agree it seems similar to #3866.

It is intermittent. And weirdly, given enough time (30m), it the exec probe eventually succeeded.

Other notes, the actual resource usage was nowhere near CPU or memory limits of the node. It was one of the first thing we checked since based on past experience, exec probes can easily fail if the node ran out of CPU cycles to actually execute the probe in a timely manner.

from bottlerocket.

Likely this: containerd/containerd#10036

Let's try the repro case but 1.19.3 included the containerd 1.6.30 update so it lines up.

from bottlerocket.

Resolving now that 1.19.4 is out.

from bottlerocket.