This sample will show you how to build a serverless data lake on AWS with minium effort to build and connect the workflow. Includes topics as following:
- Data Ingestion with Kinesis Firehose
- AWS Glue for data catalog
- Amazon Athena to query and analyze data on S3
Kinesis Data Firehose delivery streams continuously collect, transform, and load streaming data into the destinations that you specify.
Kinesis -> Kinesis Firehose -> create delivery stream
Chose S3 as your target, and provide your S3 bucket name, and prefix
Your delivery stream created successfully
Click it to detail information page
Wait for 5 min and go to S3 bucket and check data is generated and load to S3
The AWS Glue Data Catalog is an index to the location, schema, and runtime metrics of your data. You can use a crawler to populate the AWS Glue Data Catalog with tables. This is the primary method used by most AWS Glue users.
Glue -> Crawler -> Add Crawler
Includes S3 path you want to crawl
Configure your output database
Check crawler is created, and run it!
Crawler completed and made the following changes.
Check table is created in database
Check detail information in table
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.
Athena -> chose database which you created table