Giter VIP home page Giter VIP logo

security-data-lake-kinesis's Introduction

AWS Security Data Lake Log Transformation using Kinesis Data Analytics


This guide runs through how a data streaming pipeline that enriches and transforms ingested logs can be deployed in AWS using Kinesis and Glue services.
In this PoC, a python script is used to generate logs which is ingested by a Kinesis Data Stream. The logs simulated contains a “port_number” field. Kinesis Data Analytics transforms the log data in the Kinesis Data Stream and inserts the curated logs into another Kinesis Data Stream. The curated logs will be enriched with a “tag” field with its value dependent on the value of the “port_number” field. Kinesis Firehose Delivery Stream ingests the data from the Kinesis Data Stream for curated logs and streams it into an S3 bucket. The data is partitioned in the S3 bucket by year, month, day and hour.
A detailed walkthrough of the deployment steps can be found here (https://quip-amazon.com/9y2SAbT7CWRS#XQA9AAC7E4R).



CloudFormation Resources

Logical ID Type
AnalyticsApplication AWS::KinesisAnalyticsV2::Application
AnalyticsServiceExecutionRole AWS::IAM::Role
AthenaWorkgroup AWS::Athena::WorkGroup
FirehoseServiceExecutionRole AWS::IAM::Role
FirehoseStream AWS::KinesisFirehose::DeliveryStream
GlueDatabase AWS::Glue::Database
GlueTable AWS::Glue::Table
InputStream AWS::Kinesis::Stream
OutputStream AWS::Kinesis::Stream
S3Logs AWS::S3::Bucket

security-data-lake-kinesis's People

Contributors

awsryancwj avatar

Watchers

Kostas Georgiou avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.