Comments (16)
Hi,
I confirm I still got the issue:
$ kubectl describe pod/kube-slack-78b78b89cf-jh9dx
Name: kube-slack-78b78b89cf-jh9dx
Namespace: kube-system
Node: node6/1.1.1.1
Start Time: Sun, 11 Mar 2018 18:57:01 +0100
Labels: app=kube-slack
pod-template-hash=3463464579
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-slack-78b78b89cf","uid":"5c5727a2-2555-11e8-9b5d-0007cb...
scheduler.alpha.kubernetes.io/critical-pod=
Status: Running
IP: 10.233.68.25
Controlled By: ReplicaSet/kube-slack-78b78b89cf
Containers:
kube-slack:
Container ID: docker://ab1f8a71bc1e303327e4c8edfd38a0bde66fbd7ce18e7e175c70d36547706ce1
Image: willwill/kube-slack:v3.2.3
Image ID: docker-pullable://willwill/kube-slack@sha256:dbfc705ba68b7079ada1e913250d76c50754ef3c211339e900b3a2dacb2c2a0b
Port: <none>
State: Running
Started: Sun, 11 Mar 2018 18:57:11 +0100
Ready: True
Restart Count: 0
Environment:
SLACK_URL: https://hooks.slack.com/services/xxx/yyy
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-kg9rw (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-kg9rw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-kg9rw
Optional: false
QoS Class: BestEffort
Node-Selectors: node-role.kubernetes.io/node=true
Tolerations: <none>
Events: <none>
My message when deleting a pod from a daemonset for example:
containers with unready status: [traefik]
kube-system/traefik-2qzq7: ContainersNotReady
containers with unready status: [traefik]
If you want to know more about what the daemonset looks like: https://github.com/MySocialApp/kubernetes-helm-chart-traefik/blob/master/kubernetes/templates/daemonset.yaml
Thanks
from kube-slack.
Are your pod showing as 0/1 in kubectl get pods? (or something like 2/3 if you use multi containers pod)
Usually this is caused by having a readiness check on a pod that is slow to boot. We're having this issue as well (our monolithic can take like 5 min to boot) It's on my radar to search for a solution to this but it could be around the second half of the year as we're busy with upcoming product launches.
If your container is showing as 1/1 and you still get ContainersNotReady then it's probably a bug. Please comment with details if this is the case.
from kube-slack.
I think I know the problem. Should be a quick fix. I'll try testing it.
from kube-slack.
Hi,
I confirm I get 1/1 or 2/2 or 3/3. It happens while booting. The readiness check is always working, I tried to delay it as max as possible but this get the message in the early boot. It's like if the Container Not Ready was catch before the readiness check happened
from kube-slack.
I added ContainersNotReady to the blacklist (plus Longnotready bugfix). Could you please test that the patch is working? The image is tagged latest on Docker Hub.
from kube-slack.
good for me ! thanks
from kube-slack.
Released as v3.2.3
from kube-slack.
Sorry but I still got the issue
from kube-slack.
https://github.com/wongnai/kube-slack/blob/master/src/monitors/waitingpods.js#L11 ContainersNotReady is blacklisted. Could you please give
- Docker tag you are using
- Slack message (feel free to censor the pod/container name) as I need to know whether it is trigged by waitingpods or longnotready.
from kube-slack.
I'm going to test with v3.2.3, I tested with latest when you asked for test. I'll keep you updated
from kube-slack.
So it is indeed triggered by LongNotReady:
kube-slack/src/monitors/longnotready.js
Line 74 in abfb14a
I'm taking vacation next week, so I might be able to work on this around end of the month. Sorry for the wait.
from kube-slack.
Could you please try the following?
- Set initialDelaySeconds back to default value. Setting this value cause pods to be not ready longer.
- Set the
NOT_READY_MIN_TIME
environment variable to kube-slack. The default is 60000 which is 60s. In our production system we use 300000 as we have a Java application that is very slow to boot.
from kube-slack.
Hi, that doesn't change any thing. Even very small software that runs in a few seconds have the same issue.
from kube-slack.
I just tested this in our cluster and I still can't reproduce:
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- image: kitematic/hello-world-nginx
name: hello-world-nginx
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /
port: 80
- NOT_READY_MIN_TIME=300000
- kube-slack 3.2.2 (image ID docker-pullable://willwill/kube-slack@sha256:6213f0e4bf3258b4265bea1a5706f87c2477dc8ba29905ed865efa18dbdbfa9f)
- Kubernetes v1.7.8 (I haven't test, but we haven't get false positive on our v1.9.4-gke.1 cluster for some times as well)
from kube-slack.
Hi,
Sorry for the late answer, I'm currently testing with the latest version and your suggestion on NOT_READY_MIN_TIME.
Thanks
from kube-slack.
Looks good ! Thanks
from kube-slack.
Related Issues (20)
- Drain support HOT 1
- Proxy setting to reach slack webhooks. HOT 10
- Use kubernetes-client promise
- Add support of global kube-slack channel configuration. HOT 5
- Sometimes PostStartHookError message is "" even though the event has a error log HOT 2
- Feature Request - Option of Slack Username HOT 6
- Getting cert error when pod starting HOT 2
- kube-slack can mix-up different pods if they have same label HOT 2
- "Container entered status *Running*" in the wrong channel HOT 3
- Lots of errors when upgrading to 4.0.1 HOT 5
- Logs are not forwarded by fluent-bit by default HOT 5
- Posting to different channels per pod with newer Slack APIs HOT 2
- helm chart not upto date HOT 1
- feature request: to watch only pods with set pod annotation
- Feature Request: Allow to configure the different username for slack notifications
- No Message during Deployment
- Directed Slack Notifications
- Configuration property "slack_username" is not defined HOT 2
- kube-slack/ignore-pod annotation does not work if you set an empty string as value.
- Pod Security Policy compliant: runAsNonRoot HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kube-slack.