cloudreach / aws-lambda-es-cleanup Goto Github PK

View Code? Open in Web Editor NEW

114.0 15.0 70.0 145 KB

AWS Elasticsearch Lambda Curator

Home Page: https://www.cloudreach.com/blog/elasticcleaner-for-aws-elasticsearch/

License: Other

Python 36.12% HCL 59.44% Shell 4.43%

aws elasticsearch curator python lambda

aws-lambda-es-cleanup's Introduction

AWS Lambda Elasticsearch Index Cleanup

Overview

This AWS Lambda function allows you to delete the old Elasticsearch indexes using SigV4Auth authentication. You configure the AWS Elasticsearch Access Policy authorizing the Lambda Role or the AWS Account number instead of using the IP address whitelist.

Diagram

Getting Started

How To install

Clone your repository

$ git clone [email protected]:cloudreach/aws-lambda-es-cleanup.git
$ cd aws-lambda-es-cleanup/

Configure in a proper way the IAM policy inside json_file/es_policy.json and json_file/trust_policy.json

Create the IAM Role

$ aws iam create-role --role-name es-cleanup-lambda \
	--assume-role-policy-document file://json_file/trust_policy.json

$ aws iam put-role-policy --role-name es-cleanup-lambda \
    --policy-name es_cleanup \
    --policy-document file://json_file/es_policy.json

if your lambda is running inside the VPC also attach the these policies

arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole

Create your Lambda package

$ zip es-cleanup-lambda.zip es_cleanup.py

Lambda deployment

Using awscli you can create your AWS function and set the proper IAM role with the right Account ID

$ export AWS_DEFAULT_REGION=eu-west-1
$ ESENDPOINT="search-es-demo-zveqnhnhjqm5flntemgmx5iuya.eu-west-1.es.amazonaws.com" #ES endpoint

$ aws lambda create-function \
	--function-name es-cleanup-lambda \
	--environment Variables={es_endpoint=$ESENDPOINT} \
	--zip-file fileb://es-cleanup-lambda.zip \
	--description "Elasticsearch Index Cleanup" \
	--role arn:aws:iam::123456789012:role/es-cleanup-lambda \
	--handler es_cleanup.lambda_handler \
	--runtime python3.6 \
	--timeout 180

If you want to send variables and not to use environment

$ export AWS_DEFAULT_REGION=eu-west-1

$ aws lambda create-function \
	--function-name es-cleanup-lambda \
	--zip-file fileb://es-cleanup-lambda.zip \
	--description "Elasticsearch Index Cleanup" \
	--role arn:aws:iam::123456789012:role/es-cleanup-lambda \
	--handler es_cleanup.lambda_handler \
	--runtime python3.6 \
	--timeout 180

Lambda invoke with parameters

is it possible to override the default behaviour passing specific payload

$ aws lambda invoke
 --function-name es-cleanup-lambda \
 outfile --payload \
 '{"es_endpoint":"search-es-demo-zveqnhnhjqm5flntemgmx5iuya.eu-west-1.es.amazonaws.com"}'

Create your AWS Cloudwatch rule:

$ aws events put-rule \
	--name my-scheduled-rule \
	--schedule-expression 'cron(0 1 * * ? *)'


$ aws lambda add-permission \
	--function-name es-cleanup-lambda \
	--statement-id my-scheduled-event \
	--action 'lambda:InvokeFunction' \
	--principal events.amazonaws.com \
	--source-arn arn:aws:events:eu-west-1:123456789012:rule/my-scheduled-rule


$ aws events put-targets \
	--rule my-scheduled-rule \
	--targets file://json_file/cloudwatch-target.json

Lambda configuration and OS parameters

Using AWS environment variable you can easily modify the behaviour of the Lambda function

Variable Name	Example Value	Description	Default Value	Required
es_endpoint	search-es-demo-zveqnhnhjqm5flntemgmx5iuya.eu-west-1.es.amazonaws.com	AWS ES fqdn	`None`	True
index	`logstash*`	Index/indices to process comma separated, with `all` every index will be processed except the one listed in `skip_index`	`.*`	False
skip_index	`.kibana,.kibana_5`	Index/indices to skip	`.kibana*`	False
index_format	`%Y.%m.%d`	Combined with `index` varible is used to evaluate the index age	`%Y.%m.%d`	False
delete_after	`7`	Numbers of days to preserve	`15`	False

Serverless Framework

Editing the file serverless.yml, you can deploy your function in AWS using Serverless Framework

$ git clone [email protected]:cloudreach/aws-lambda-es-cleanup.git
$ cd aws-lambda-es-cleanup/
$ serverless deploy
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading function .zip files to S3...
Serverless: Uploading service .zip file to S3 (7.13 KB)...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
......................
Serverless: Stack update finished...
Service Information
service: es-cleanup-lambda
stage: prod
region: eu-west-1
api keys:
  None
endpoints:
  None
functions:
  es-cleanup-lambda: es-cleanup-lambda-prod-es-cleanup-lambda

Terraform deployment

This lambda function can be also build using terraform followings this README.

How to Contribute

We encourage contribution to our projects, please see our CONTRIBUTING guide for details.

License

aws-lambda-es-cleanup is licensed under the Apache Software License 2.0.

Thanks

Keep It Cloudy (@CloudreachKIC)

aws-lambda-es-cleanup's People

Contributors

Stargazers

Watchers

aws-lambda-es-cleanup's Issues

Fix terraform security group

Current security group doesn't allow DNS requests, therefor lambda function can't resolve ES cluster DNS name.

('Connection aborted.', gaierror(-3, 'Temporary failure in name resolution')): ConnectionError
Traceback (most recent call last):
File "/var/task/es-cleanup.py", line 180, in lambda_handler
for index in es.get_indices():
File "/var/task/es-cleanup.py", line 165, in get_indices
return self.send_to_es("/_cat/indices")
File "/var/task/es-cleanup.py", line 119, in send_to_es
res = session.send(preq)
File "/var/runtime/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/var/runtime/botocore/vendored/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-3, 'Temporary failure in name resolution'))

skip_index parameter to support regular expressions

Taking the following scenario into discussion where we would have the ES cleaner run against all indices in an ES cluster with the exception of a particular indice pattern, for example to exclude all indices matching openshift-infra-%Y.%m.%d.

I've tested adding an expression as openshift-infra-* to the skip_index parameter and when invoked manually, it finds the respective indice but also returns:

"Keeping index: openshift-infra-2019.10.01"

For the default .kibana,.kibana_5 indice that "Keeping..." output is not returned, thus I assume it is skipped successfully.

Not very familiar with Python, thus I want to ask if the skip_index variable supports delimiting indices through regular expressions? Alternatively I guess the indices can be defined in the index variable instead of all value, which also raises the same question, if it supports expressions? :)

Lambda Function Times Out When Invoked Within AWS

When I run es-cleanup.py locally it deletes the indices as expected.

When I run 'Test' in Lambda or lambda invoke on the command-line I get a timeout and the operation doesn't complete. CloudWatch Logs simply shows the time the command was invoked but no errors.

What could be wrong here?

create serverless configuration for AWS ES running on VPC

only available in terraform #10

Cleanup lambda deletes .kibana_N indices

From v6.7.0 of Kibana, Kibana settings are stored within the .kibana_N indices where the .kibana index is an alias pointing to the latest .kibana_N index (e.g. .kibana_5).

Rather than only ignoring the .kibana index, it would make sense to ignore anything starting with .kibana

Evaluate more than one index format

In my AWS Open Search cluster I have diverse index formats, such as:

xxxxx-2021.09.09 (%Y.%m.%d index format)
xxxxx-2021-08-11 (%Y-%m-%d index format)

Will be great if the lambda could evaluate different index formats simultaneously, i.o index_format = [ %Y.%m.%d, %Y-%m-%d, .... ]

Code optimisation when aws_lambda_function.vpc_config is solved

when the related issue of

hashicorp/terraform-provider-aws#443
already resolved in
hashicorp/terraform-provider-aws#1341
but never merged

Optimised the terraform code

Create snapshots before deleting indexes

This would allow having a "cold" storage of logs/indexes before destroying them completely. The snapshots themselves should have a retention time as well.

ValueError: You can only send PreparedRequests.

Hi, I'm using in lambda function. From few days I'm facing below issue.

You can only send PreparedRequests.: ValueError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 180, in lambda_handler
for index in es.get_indices():
File "/var/task/lambda_function.py", line 165, in get_indices
return self.send_to_es("/_cat/indices")
File "/var/task/lambda_function.py", line 119, in send_to_es
res = session.send(preq)
File "/var/runtime/botocore/vendored/requests/sessions.py", line 551, in send
raise ValueError('You can only send PreparedRequests.')
ValueError: You can only send PreparedRequests.

send_error is never triggered

The sns_alert is not used anywhere.

update readme to deploy in vpc with CLI

as discussed in #20 add on readme add AWSLambdaVPCAccessExecutionRole

Delete by Mapping

Sorry, this is an enhancement!

aws-lambda-es-cleanup works when the date is part of the index like cwl-2017.08.28.

I'm wondering how challenging it would be to delete by date as part of a mapping? In the following example, the date is part of the index.

{
  "mappings": {
    "my_type": {
      "properties": {
        "date": {
          "type":   "date",
          "format": "yyyy-MM-dd"
        }
      }
    }
  }
}

CFN

Thank you for your response regarding me earlier issue. I agree that my request was too customized to offer any benefit.

If it's of interest, I built a CFN for this at, https://github.com/piedmont01/aws-lambda-es-cleanup/blob/master/CFN/aws-lambda-es-cleanup.yaml

Erick Kendall

ARN

If this ARN is valid this should be parameterised/omitted.

terraform providor / lambdas created incorrectly

In the Terraform Provider, you are creating a package, however said package is not creating the file as other readable. This results in the function being unable to execute and spewing out an error message from the Lambda function:
START RequestId: x-x-x-x Version: $LATEST
module initialization error: [Errno 13] Permission denied: '/var/task/es-cleanup.py'

As per docs here, The file needs to be world readable as per the requirements:
https://docs.aws.amazon.com/lambda/latest/dg/deployment-package-v2.html

Further to this, this is actually a bug either with the data provider or lambda uploader in terraform:
hashicorp/terraform-provider-aws#3918

Zip info confirming permissions issue with world readable file:
$zipinfo es-cleanup.zip
Archive: es-cleanup.zip
Zip file size: 2346 bytes, number of entries: 1
-rwxrwx--- 2.0 unx 6485 bl defN 49-Jan-01 00:00 es-cleanup.py
1 file, 6485 bytes uncompressed, 2206 bytes compressed: 66.0%