Switch to ILM instead of rollover about homeassistant-elasticsearch HOT 16 CLOSED

I'd honestly love to remove the rollover code that we currently have, as it's rather clunky. OTOH, forcing users to upgrade to 6.7 isn't necessarily ideal either.

Any thoughts?

👍 : Rip out rollover
👎 : Keep it in!

from homeassistant-elasticsearch.

@markwalkom (&others): I've published version 0.2.0-BETA-4 which adds support for ILM, and drops support for the manual rollover process. As such, the available configuration options have updated too. The feat/ilm branch has an updated readme outlining the new options:

https://github.com/legrego/homeassistant-elasticsearch/blob/feat/ilm/README.md#advanced-configuration

This version has also been published as a beta release on HACS if that's easier for you too.
Looking forward to your feedback!

from homeassistant-elasticsearch.

@dancwilliams can you try the v0.2.0-BETA-4 release and let me know if that fixes the issue? It appears resolved on my side, but then again, I missed it the first time around 😄

from homeassistant-elasticsearch.

@legrego Looks like it worked this time. All pieces are in place and no errors.

from homeassistant-elasticsearch.

I just did the update worked flawlessly ! thanks !

from homeassistant-elasticsearch.

Thanks for the issue @dsztykman! This is something I've wanted to do for a while, so now that ILM is publicly available, we can get started on this 🎉

from homeassistant-elasticsearch.

Very selfish from me but let's just rip out rollover 👍

from homeassistant-elasticsearch.

To keep things simple, I was thinking of starting off with a simple hot / delete setup.

Maybe something like:

# configuration.yml
elastic:
  url: '....'
  ilm:
    enabled: True # (will detect if available via license)
    hot_max_size: 30GB
    hot_max_age: #(Default None)
    delete_min_age: 356d #(Default None)

Would build a policy similar to the following:

PUT _ilm/policy/hass_policy   
{
  "policy": {                       
    "phases": {
      "hot": {                      
        "actions": {
          "rollover": {             
            "max_size": "30GB"
          }
        }
      },
      "delete": {
        "min_age": "365d",           
        "actions": {
          "delete": {}              
        }
      }
    }
  }
}

Thoughts on this?

from homeassistant-elasticsearch.

Any volunteers to beta-test this feature? I'm hoping to have something ready next week, but there's only so much testing I can do against my own installation.

from homeassistant-elasticsearch.

@legrego, I know it's been a while, if you still need someone lemme know

from homeassistant-elasticsearch.

Thanks @markwalkom, I definitely will need a tester or two, but I've stalled on this recently, and need to tie up some loose ends before I have anyone run it. I'll let you know once I'm ready!

from homeassistant-elasticsearch.

@legrego I just installed the beta and so far so good! I will let you know as things progress.

from homeassistant-elasticsearch.

@legrego I am seeing the new index being created, hass-events-v4-000001, and the new index template, hass-index-template-v4, but I do not see the lifecycle policy being created. Looks like it is failing full setup with the following error:

2020-02-19 16:52:13 ERROR (MainThread) [homeassistant.setup] Error during setup of component elastic
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 170, in _async_setup_component
    hass, processed_config
  File "/config/custom_components/elastic/__init__.py", line 95, in async_setup
    index_manager.setup()
  File "/config/custom_components/elastic/__init__.py", line 272, in setup
    self._create_ilm_policy(self._config)
  File "/config/custom_components/elastic/__init__.py", line 352, in _create_ilm_policy
    encoded_policy_name = urllib.quote(
AttributeError: module 'urllib' has no attribute 'quote'

from homeassistant-elasticsearch.

I am running ES version 7.5.1 on an Ubuntu 18.04 box. I am running the latest and great HASSOS and all that.

from homeassistant-elasticsearch.

Thanks for the quick feedback @dancwilliams! Looks like my last minute change to encode the url properly is failing. I’ll try to get a revert up soon to see if that fixes this issue.

Edit: my inexperience with Python is showing. Looks like I imported the module incorrectly for Python 3

from homeassistant-elasticsearch.

FYI to beta testers: I will (eventually) be combining the following PRs into the next 0.2.0 release:

• #83
• #85
• #84

If you're beta-testing this specific feature, then you will need to manually adjust your index mapping to incorporate the new fields introduced in #83. Alternatively, you can drop the current hass-events-v4 based indices and template prior to updating, if you don't care about the data.

from homeassistant-elasticsearch.