This sample will show you how to build a serverless data lake on AWS with minium effort to build and connect the workflow. Includes topics as following:

1. Data Ingestion with Kinesis Firehose
2. AWS Glue for data catalog
3. Amazon Athena to query and analyze data on S3

Kinesis Data Firehose delivery streams continuously collect, transform, and load streaming data into the destinations that you specify.

Kinesis -> Kinesis Firehose -> create delivery stream

Enter Delivery stream name

Chose S3 as your target, and provide your S3 bucket name, and prefix

Your delivery stream created successfully

Click it to detail information page

Test it with demo data

Sending demo data

Wait for 5 min and go to S3 bucket and check data is generated and load to S3

The AWS Glue Data Catalog is an index to the location, schema, and runtime metrics of your data. You can use a crawler to populate the AWS Glue Data Catalog with tables. This is the primary method used by most AWS Glue users.

Glue -> Crawler -> Add Crawler

Provide Crawler name

Includes S3 path you want to crawl

Choose an IAM role

Configure your output database

Check crawler is created, and run it!

Crawler completed and made the following changes.

Check table is created in database

Check detail information in table

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.

Athena -> chose database which you created table

Run SQL "SELECT * FROM "sampledb"."raw" limit 10;"

See results