Support Elastic Common Schema (ECS) about homeassistant-elasticsearch HOT 18 CLOSED

@sejnub Thanks for your consideration. @legrego and I both happen to be employed by Elastic, so we're kind of familiar with ECS 😄

While ECS is in beta, we have been unsure how to proceed, and don't want to do something that we might have to walk back in the future. That's where much of our commentary has been centered: not having to back it out and change it at some future point because we leaped too soon.

We welcome your ideas and comments on how to address this. We're not settled on a course of action yet, so your thoughts are appreciated.

from homeassistant-elasticsearch.

I threw together a PR as a starting point for discussions. Please feel free to review and comment on the approach: #56

from homeassistant-elasticsearch.

I was just going to add this, but you beat me to it!

Need any help?

from homeassistant-elasticsearch.

@untergeek I'll take all the help I can get! If you want to contribute a PR, or even just a list of mappings here, that would be awesome. I don't expect that I'll have time for another week or so

from homeassistant-elasticsearch.

Roger that. I would love to have this ECS compliant before I deploy this to my HA environment.

from homeassistant-elasticsearch.

Would you like to see dynamic templates in the mapping? Such that float is float or half-float, strings are keyword, etc.? On or off is bool type?

from homeassistant-elasticsearch.

I'm certainly not opposed -- I honestly wasn't aware of dynamic templates when I first wrote this, and I'm still not that well-versed in them. If it makes it easier or more reliable to support ECS, then I say go for it 👍

from homeassistant-elasticsearch.

Brain dump of proposed mappings, based on the most current stable beta (v1.0.0-beta1 as of the time of this writing). This is not an exhaustive list.

I haven't found a place for the "dumping ground" of fields that we publish under the attributes field today. attributes contains all of the, well, entity attributes that HASS records. Each entity can have a specific set of attributes that are not known ahead of time, so we need a way to dynamically create these, while being compliant with ECS.
ECS allows for custom fields, so long as we follow the published guidelines.

Base Fields

@timestamp - The timestamp of when the hass event takes place.
tags - List of keywords to tag each event. Ideas of what to include here?
labels - Key/value pairs. Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword. Ideas of what to include here?
message - Probably not needed when capturing event data

Agent Fields

agent.version - should correspond to the installed HASS version.
agent.name - should be derived from the HASS discovery info. Maybe something like home-assistant: ${location_name}?
agent.type - Should be something like hass, home-assistant, home assistant.
agent.id = Maybe this is the external HASS url?

ECS Fields

ecs.version - should correspond to the supported ecs version

Event Fields

I had high hopes for this category, but we might not have a use-case for event fields.

event.id - I don't believe HASS provides one, so maybe this is something we derive from the @timestamp + entity_id ?
event.category - Not sure what to put here. Maybe omit, or something fixed like state_change?
event.type - Also not sure what to put here, if anything.

Geo Fields

geo.location - should be populated similar to how attributes.es_location is populated today

Host Fields

These should optionally be included by configuration setting, if the user chooses to publish them (default = true). If opted-in, I believe all fields defined here should be sent.

from homeassistant-elasticsearch.

These look good. Thoughts as follows:

• All of the agent fields should be derived from the HASS discovery info:
  • Why not put the HASS discovery base_url in as url fields?
  • version => agent.version
  • location_name => agent.name
  • version => agent.version
  • Mapping agent.type firmly to hass or home-assistant makes sense.
• Do we need event.id at all? Just a unique identifier? Why wouldn't the auto-generated _id be sufficient? If HASS ever provides one, then it makes sense to add it, but otherwise unnecessary IMO.
• event.category or event.type does indeed sound good as a place to identify the type as state_change

from homeassistant-elasticsearch.

Could one of you publish the mapping template that you currently use for your hass indices. I am just beginning with elasticsearch and you probably have some working mapping even if currently uses the not ECS conform field names. I think this mapping template should be part of this repository. I offer to help improving it.

from homeassistant-elasticsearch.

Thanks for the feedback @untergeek!

Why not put the HASS discovery base_url in as url fields?

Good idea, I hadn't considered that.

Do we need event.id at all? Just a unique identifier? Why wouldn't the auto-generated _id be sufficient? If HASS ever provides one, then it makes sense to add it, but otherwise unnecessary IMO.

We likely don't need an event.id here, you make a good point.

Any thoughts on the attributes dumping ground? We don't have a way of getting a definitive list of these, as every HASS installation will be different. We can easily move any geo fields into device.geo, but the rest would be difficult. I'm specifically concerned about ECS's guidelines for writing/naming fields.

We could omit/replace special characters in field names if needed, but otherwise, I don't think we can't really control the naming of fields.

from homeassistant-elasticsearch.

Hey @sejnub,

The index template is created within elastic.py. You shouldn't need to craft one by hand for this component to work. I'd welcome your help if you're interested in improving it!

Python is not my strong language, so there's likely opportunities for cleanup as well.

from homeassistant-elasticsearch.

@legrego
Ups sorry, I didn't know that the index template is already there. I will try to find out, why it wasn't used on my elk instance. On my instance elasticsearch tried to guess the mapping and now handles the field "value" as type "long", which isn't so nice for thermostats.

Thanks!

from homeassistant-elasticsearch.

@legrego For anything not specified in ECS, we could make a hass root-level object in the mapping, and put stuff beneath that. It would therefore not stomp on any pre-existing ECS field names, and would therefore not pose a horrible problem if some of the field names are not 100% ECS compliant.

from homeassistant-elasticsearch.

For anything not specified in ECS, we could make a hass root-level object in the mapping, and put stuff beneath that. It would therefore not stomp on any pre-existing ECS field names, and would therefore not pose a horrible problem if some of the field names are not 100% ECS compliant.

I like that approach, and it seems like a fair tradeoff for our needs!

from homeassistant-elasticsearch.

There is a nice quite new video about the ecs: https://www.youtube.com/watch?v=XNeTAHlimQI

from homeassistant-elasticsearch.

We are probably safe to take a closer look at this now that ECS has turned 1 🎉

from homeassistant-elasticsearch.

I’ll leave that PR open for comments until March 24th. Barring any feedback, I’ll plan on merging it shortly thereafter.

from homeassistant-elasticsearch.