Comments (13)
@jtlyk thank you for the update. I’m currently using the « cluster monitoring operator » provided with OpenShift so I can’t use jsonnet to disable the rule. The only thing I can do for now is to completely disable « etcd monitoring ». Or maybe the cluster-monitoring operator will update itself and will take into account the modification ??
from cluster-monitoring-operator.
To be more specific the query regarding this alert always retrieve "100" as a value. I suspect that the query is false.
Current query is :
100 * sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{grpc_code!="OK",job=~".*etcd.*"}[5m])) / sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) > 5
in my opinion it should be (remove *100 at the begining) :
sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{grpc_code!="OK",job=~".*etcd.*"}[5m])) / sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) > 5
from cluster-monitoring-operator.
I also see this on OpenShift 3.11 after enabling etcd monitoring.
from cluster-monitoring-operator.
The current query seems correct to me. It alerts on more than 5% of requests failing.
On my personal cluster I can see that there's a watch method pending for 4min.
{grpc_method="Watch",grpc_service="etcdserverpb.Watch",instance="10.135.73.45",job="etcd"}
Is it the same for you? Maybe we should ignore the watches here.
from cluster-monitoring-operator.
Tested your query, result is empty. For me the strange thing is that I have the same alert on all the OpenShifts Cluster I have installed. and the query result for EtcdHighNumberOfFailedGRPCRequests gives alway a value of 100
from cluster-monitoring-operator.
Hi @nabbdl just wondering if you still seeing that error and if you resolve the mystery? I'm facing the same error here and not sure why is happening.
from cluster-monitoring-operator.
Hi @zot24. Unfortunately, I'm still seeing the same error.
from cluster-monitoring-operator.
@nabbdl jtlyk I have been doing some research and after reading a lot of comments I think I'll just ignore those alert for now poseidon/typhoon#175 there are a bunch of issues regarding this error message, but pretty much what's going on it's summarized in this issue etcd-io/etcd#10289 and there is still not a fix for it.
In more detail I think this is the offended line https://github.com/gyuho/etcd/blob/0cf9382024da6132cb5f0778c3fb43e4a6c88afd/etcdserver/api/v3rpc/util.go#L111
from cluster-monitoring-operator.
If you using jsonnet you could add the following to suppress that rule for now:
{
prometheusAlerts+:: {
groups: std.map(
function(group)
if group.name == 'etcd' then
group {
rules: std.filter(
function(rule)
rule.alert != 'etcdHighNumberOfFailedGRPCRequests',
group.rules
),
}
else
group,
super.groups
),
},
}
from cluster-monitoring-operator.
@nabbdl jtlyk this just got merge #340
from cluster-monitoring-operator.
You could do what I did in my clusters and create a silence for those alerts in alertmanager, but it does look like they may be backporting it currently: #383
from cluster-monitoring-operator.
Fix backported in #383
/close
from cluster-monitoring-operator.
@paulfantom: Closing this issue.
In response to this:
Fix backported in #383
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from cluster-monitoring-operator.
Related Issues (20)
- {Request} Ability to configure Prometheus Exporters HOT 5
- Invalid externalURL since ad2d747f2ef037647ebc9f7d5b22118d0dab4bd2 HOT 4
- Release 4.12 Checklist HOT 3
- Missing Metrics in Prometheus on openshift 3.11 HOT 4
- User workload prometheus scraps targets managed by monitoring prometheus instance HOT 4
- Release 4.13 Checklist HOT 5
- crio metrics via http is not safe HOT 7
- Allow kube-state-metrics configuration HOT 3
- Reject invalid PrometheusRule objects beforehand rather than failing the reconciliation HOT 5
- Alerts generated by user-workload-monitoring are not included in remoteWrite HOT 4
- typo in 4.14 CHANGELOG.md and CONTRIBUTING.md HOT 3
- OpenTelemetry Support HOT 16
- kubelet metrics not working in after upgrade to OKD 4.13 HOT 12
- wrong annotation for ThanosRulerConfig.Resources HOT 1
- I can't increase Prometheus startupProbe failureThreshold value HOT 6
- user-workload-monitoring doesn't support honoring labels HOT 4
- Additional Scrape Configuration is not supported HOT 8
- How to change default servicemonitors interval HOT 5
- Support enableRemoteWriteReceiver flag in prometheus{,K8s} config HOT 6
- Future Release Branches Frozen For Merging | branch:release-4.16 branch:release-4.17
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluster-monitoring-operator.