Tracking ES node types about elasticsearch_exporter HOT 26 CLOSED

That's what I had in mind, but I'd like to hear what @zwopir thinks about that.

from elasticsearch_exporter.

thanks for your help! I just need an approval from my collegue for this one to get it merged into master. I'll then create a release candidate

from elasticsearch_exporter.

just pushed the image

from elasticsearch_exporter.

Sounds reasonable to have this metric one way or the other.

from elasticsearch_exporter.

Or may be even better to expose two dimentions: esdata=true/false esmaster=true/false, because this is what ES actually exposes, and people may have their own non-standard setups.

from elasticsearch_exporter.

We should expose the different modes a node can have, like node_data, node_master and maybe node_ingest, in different labels.

from elasticsearch_exporter.

Do you mean something like elasticsearch_indices_request_cache_evictions{cluster="elasticsearch", host="10.1.1.1", name="esdata1",..., node_data="true"}?

from elasticsearch_exporter.

yes, we're having the following in the json

...
  "nodes": {
    "G65WEllISliAl4-ZA3QwdQ": {
      "timestamp": 1509098890579,
      "name": "prod-es53-3",
      "transport_address": "10.10.10.10:9300",
      "host": "10.10.10.10",
      "ip": "10.10.10.10:9300",
      "roles": [
        "master",
        "data",
        "ingest"
      ],
...

so a node can have multiple roles. As the labels shouldn't be overlapping, that's probably the best way to go. Thinks to consider:

• true/false label values are ugly
• we should agree on a set of possible roles. master, ingest, data, anything else?

from elasticsearch_exporter.

I guess if you define node.ml it would come up in roles as well.

Also for ES 2.x, there is no roles json node, but rather attributes node that looks a bit different:

            "attributes": {
                "data": "false",
                "master": "true"
            },

from elasticsearch_exporter.

thanks for the hints. Just tried out if defining ml adds a role: it doesn't, but thanks anyway!

from elasticsearch_exporter.

from elasticsearch_exporter.

no worries, I hadn't :D ...
with xpack and the default setting (ml=enabled) I get the following json

      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "attributes": {
        "ml.max_open_jobs": "10",
        "ml.enabled": "true"

so the hint was definitely helpful, especially when trying to be compatible with ES 2.x and evaluation attributes in the node json

from elasticsearch_exporter.

Ok, so we have master, data, ingest, and ml roles and a node can assume several roles. Since P8s doesn't support lists for label values AFAIK, do you have any better idea besides true/false labels?

from elasticsearch_exporter.

you're right, p8s doesn't support label lists. So true/false labels are the way to go. As a first version I would ignore the attributes.ml.enabled and just create the true/false labels es_master_node, es_data_node, es_ingest_node.

from elasticsearch_exporter.

Great. Can't wait to test it :)
Do you plan to support ES 2.x in your first version? I have both 5.x and 2.x deployed and it will be great to have it.

from elasticsearch_exporter.

I'm having a look this moment. Can't find the attributes field in the json right now. What API endpoint did you use? We are creating our test json by starting a docker node
docker run -p 9200:9200 elasticsearch:2.4.5 and then curl localhost:9200/_nodes/stats

what 2.x version do you use?

from elasticsearch_exporter.

I'm on 2.3.5 and querying the same URL you mention. With 2.x, they don't report attributes unless they are different from defaults, which are master=true, data=true. That's why you don't see anything by default, and also a reason why my data node in the example below reports just master=false.

$ curl -s localhost:9200/_nodes/stats
....
    "nodes": {
        "BTMZ3IRgS7u4jFGYcn3zIA": {
            "timestamp": 1509107863017,
            "name": "esxxxxxxxxxxxxxxxxxxxxxxx",
            "transport_address": "xxxxxxxxxxxxxxx",
            "host": "xxxxxxxxxx",
            "ip": [
                "xxxxxxxxxx",
                "NONE"
            ],
            "attributes": {
                "master": "false"
            },

from elasticsearch_exporter.

ah, thanks!

from elasticsearch_exporter.

@haizaar do you know how to build go projects and give https://github.com/zwopir/elasticsearch_exporter/tree/feature/add_role_labels
a try on your 2.x? That would be nice

from elasticsearch_exporter.

@zwopir ,
I've built and run your branch. Unfortunately it does not produce good results for 2.x:

elasticsearch_breakers_estimated_size_bytes{breaker="parent",cluster="es",es_data_node="true",es_ingest_node="true",es_master_node="true",host="xxxx",name="esdata-xxxx"} xxxx
elasticsearch_breakers_estimated_size_bytes{breaker="parent",cluster="es",es_data_node="true",es_ingest_node="true",es_master_node="false",host="xxxx",name="esmaster-xxxx"} 0

You can see that data node has master=true, and master node has data=true and master=false! :)
Also there are no ingest nodes in 2.x, so I suggest detecting and removing that role for 2.x.

The attributes property for the above nodes is as follows:

# master
            "attributes": {                                                                                                    
                "data": "false",                                                                                               
                "master": "true"                                                                                               
            }

# data
            "attributes": { 
                "master": "false"                                                                                              
            },

Also I suggest to check whether http JSON node exists in node stats and set es_client_node true/false accordingly - then we'll have all of the role range covered it seems.

Thanks!

from elasticsearch_exporter.

@haizaar fixed the bug and added a new label es_client_node.

from elasticsearch_exporter.

Looks very good! The only thing is that I would remove es_ingest_node for 2.x, or at least set it to false. With your current code it's always set to true which is misleading.

from elasticsearch_exporter.

thanks for testing! Unfortunately there is no direct way of telling if we are calling a 2.x or 5.x cluster, so having es_data_node and es_master_node on 2.x and es_data_node,es_master_node and es_ingest_node on 5.x is not really possible. So we stick to the version with es_ingest_node=false on 2.x clusters.
I updated the PR (#105)

from elasticsearch_exporter.

Yeap, indeed node stats do not return version. setting it to false permanently on 2.x is very find with me - it hints that a node can't inject, and it empirically true for 2.x - in the same sense as to say that it can't make coffee, but still, true :)

I've tested your last fix and it works wonders. From my side it a big LGTM. Thank you very much for coding it quickly. When can we expect it to be released?

from elasticsearch_exporter.

here it is, @haizaar: https://github.com/justwatchcom/elasticsearch_exporter/releases/tag/v1.0.2rc2

from elasticsearch_exporter.

Great. Thanks again. When do we expect the docker image to be available at https://hub.docker.com/r/justwatch/elasticsearch_exporter/tags/?

from elasticsearch_exporter.