skip_index parameter to support regular expressions about aws-lambda-es-cleanup HOT 7 CLOSED

@EduSandu please check the new version
https://github.com/cloudreach/aws-lambda-es-cleanup/blob/e0f600561e2f0914befb2f8c236658ff47cfe8f0/es-cleanup.py

and test it.

from aws-lambda-es-cleanup.

The Regexp done like that

 self.cfg["skip_index"] = self.get_parameter("skip_index", ".kibana*") 

can match multiple patterns/indexes
I will open a new PR with the current changes

@EduSandu thanks for your support!

from aws-lambda-es-cleanup.

@EduSandu that sounds like a great idea.
a similar issue was resolved in #26 would be awesome to have some help from the community to implements regular expression for the index to skip and the index to delete.

I will try a beta version in a dedicate branch

from aws-lambda-es-cleanup.

@giuliocalzolari awesome stuff - tested OK it matched and processed all indexes using the wildcard match and also skipped the .kibana indexes.

I omitted initially during testing to change the skip_index param, which was using the comma-delimited values - the cleaner job deleted the .kibana_N indexes, but was able to restore them from a snapshot.

Wondering if the .split(",") delimeter can be kept and used to separate multiple expressions in the index & skip_index variables?
This gives more granular control and lets you to keep more important indices longer (have them skipped via the expression matching their pattern in skip_index) and drop the ones that are not that relevant earlier, in order to save disk usage.

from aws-lambda-es-cleanup.

@EduSandu uhmm very wierd...
I've setup the skip_index to also use the regexp

are you using Terraform to deploy it?
because I've realised the default value for terraform is still ".kibana" and this pattern does not match .kibana_N

can you confirm it?

from aws-lambda-es-cleanup.

@giuliocalzolari no this wasn't a new domain standed up using Terraform.
skip_index works using the regexp method, only the initial test cleared the .kibana_N indices (skip_index was still using the prior comma delimited format)

Would like to test having multiple expressions with a , delimeter, for the reasons mentioned above - guessing the following lines

from aws-lambda-es-cleanup.

#33

from aws-lambda-es-cleanup.