Giter VIP home page Giter VIP logo

wieshka / cloudfront-access-log-to-cloudwatch-metrics Goto Github PK

View Code? Open in Web Editor NEW
12.0 2.0 4.0 430 KB

CloudFront has some nice statistics burried within console, unfortunetly neither programmatical access to those data or exposure to CloudWatch is offered out of the box. Let's say you want to compliment your existing monitoring data for CloudFront distribution with some nice data which are available only within web console. This SAM Lambda will allow you to do that.

Home Page: https://discovering.technology/notes/2018/02/23/cloudfront-access-logs-to-cloudwatch-custom-metrics/

License: GNU General Public License v3.0

Python 100.00%
serverless-application-model serverless cloudwatch-metrics cloudfront aws python s3 boto3 lambda sam-lambda sam

cloudfront-access-log-to-cloudwatch-metrics's Introduction

Cloudfront access logs to CloudWatch custom metrics

CloudFront has some nice statistics burried within console, unfortunetly neither programmatical access to those data or exposure to CloudWatch is offered out of the box. Let's say you want to compliment your existing monitoring data for CloudFront distribution with some nice data which are available only within web console. This SAM Lambda will allow you to do that.

CloudFront offers to push it's accesss logs to S3 bucket of your choise, this Lamba will pick them up and convert to CloudWatch metrics.

Contents

Features

  • Currently extracts from log files count per x-edge-response-result-type - Hit / RefreshHit / Miss / LimitExceeded / CapacityExceeded / Error / Redirect count. This allows to visualize event count per second as SampleCount;
  • That should give fairly good idea how to extract other desired data from access log;

Install

  • Create S3 bucket for code or specify existing one in next step (CodeUri): aws s3 mb s3://<bucket-name-where-code-should-be-stored>. That's something you need to work normally with sam;
  • Upload code, generate CloudFormation: sam package --template-file template.yml --s3-bucket <bucket-name-where-code-should-be-stored> --output-template-file cloudformation.yml
  • Deploy CloudFormation stack: aws cloudformation deploy --template-file path/to/cloudformation.yml --stack-name <cf-stack-name> --capabilities CAPABILITY_IAM
  • Configure yout CloudFront distriubtion to write logs to just deployed S3 bucket by enabling Logging under it's generic settings
  • If all is good, you should see something like this in CloudWatch logs (for Lambda): Access log XXXXXXXXXXXXXX.2018-02-23-19.bd7ae1f4.gz originating from targetbucket-xxxxxxx with 22347 lines was parsed and pushed to CloudWatch
  • If for some reason (extending?) you need to install any additional Python packages, the absolutely easiest approach will be adding them to requirements.txt and executing: pip install -r requirments.txt -t . within repo root directory.

Note

  • This SAM template will deploy CloudFormation stack which will consist of Lambda Function and S3 Bucket + suplimentary stuff like execution roles, etc. You can find under CloudFormation Resources what exactly was deployed.
  • It is expected, that once this stack is deployed, CloudFront access logs should be configured to be pushed to created S3 Bucket.
  • Consider need to improve performance of this Lambda otherwise for very busy CloudFront distributions, you might reach timeout. Current performance is about 0.83 lines of CSV per ms aka 25 000 lines takes ~30 seconds of Lambda exec time.
  • Mind quite allowing policies given to Lambda, in more strict environment that should be modified.
  • CSV_FIELDS lists all CSV fields with it's order number and description - for your convinience to extend or if you desire to switch to csv.DictRead

Seeing is believing

Screenshot from Grafana Screenshot from CloudWatch metrics

  • normally I observe 5-15 min floating delay for data with this approach

Testing locally

  • you will need AWS SAM Local
  • sam local generate-event s3 > event.json
  • modify your event.json to match existing S3 bucket with actual access log file key
  • sam local invoke -e event.json

cloudfront-access-log-to-cloudwatch-metrics's People

Contributors

wieshka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.