This repository contains code for setting up an analytics platform on AWS that includes services as:
- AWS Managed Workflows for Apache Airflow,
- AWS Glue,
- AWS EMR,
- AWS Redshift,
- AWS S3.
Apart from the core services used, VPC network and IAM roles structure are prepared as well.
Moreover, an exemplary pipeline is defined to showcase data processing across all of the services created.
The whole project idea is an extension of Amazon MWAA for Analytics Workshop - highly recommended to go through it first if you don't feel fluent enough in AWS services and Terraform.
dags/
- Airflow DAGs definition directorydata/
- data directory for tests and/or local developmentinfra/
- whole Terraform code that creates AWS Analytics Platform infrastructure. Divides intomodules/
(services definition) as well asprod/
(production environment implementation) directories. Can be easily extended into extra environmentslogs/
- Airflow logs directorymisc/
- directory for the rest of the files. Includes:airflow_requirements.txt
- Airflow environment requirements, necessary for AWS MWAA setupemr_default_extra_parameters.json
- JSON to be passed insideemr_default
Airflow connection, contains default values for EMR jobs parameters
plugins/
- Airflow plugins directoryscripts/
- directory containing Python script files for AWS EMR and AWS Glue jobs. AWS Glue jobs has been provided separately, using only PySpark library, and creating a flow inside AWS Glue Studiosql/
- directory for SQL scripts used in AWS Redshift clusterairflow-docker-compose.yaml
- default Airflow Docker Compose yml file, extended with airflow-amazon dependency insidepyproject.toml
- repository Python environment setup file, created with poetry framework. Mainly for local development purposesREADME.md
- readme file containing whole repository documentation
Below you can find and architecture overview of all services included in the platform. It includes its all of the big blocks, omitting some details regarding exact names or specific parameters (everything can be found inside Terraform files).
The main idea behind this exemplary pipeline is to fetch data from S3, process it using AWS Glue and AWS EMR, and finally put it into Redshift. In this scenario, Glue and EMR could be joined together, as the tasks are pretty simple, but they're separate just to showcase both of the AWS functionalities inside AWS MWAA.
Below you can find a description each step separately:
s3_sensor_nyc
- sense whether there are any raw data in S3 bucketcrawl_data_nyc
- run AWS Glue Crawler to fetch metadataprocess_to_staging_nyc
- execute AWS Glue job to process raw data and put it into staging directorycreate_emr_jobflow
- starts EMR jobflow to perform data aggregation from staging directory and put it into final/gold folderadd_emr_step
- adding 2 EMR jobflow steps: put script files into HDFS and execute Spark jobemr_step_sensor
- sense whether the jobflow has been completed, so it can be terminatedterminate_jobglow
- terminate jobflow so no resources are idleload_nyc_to_redshift
- execute COPY command and load data from S3 to Redshift
To properly setup the whole platform, several actions are necessary:
- For local development:
- download aws-glue-libs Docker image (see: WS Glue local development)
- download Apache Airflow Docker image (see: Airflow local development)
- Have AWS account available to manage services described above as a part of a platform
- Go through Terraform code to alter names, account IDs to match your preference
- Execute
terraform apply
command to create AWS infrastructure - Sync directories:
dags
,plugins
,scripts
,requirements
with respective local folders and files - For exemplary pipeline usage:
- Start the pipeline in AWS MWAA environment
- Run
aws s3 cp s3://ee-assets-prod-us-east-1/modules/f8fe356a07604a12bec0b5582be38aed/v1/data/green_tripdata_2020-06.csv s3://<your-bucket>/data/raw/green/
to copy files to be sensed by S3 sensor - Monitor the rest of the pipeline's tasks executing
This is only showcase repository, please do not deploy 1:1 what you can find here! You will need to go into Terraform infrastructure setup and carefully change parameters that will suit your environment, e.g.: objects (S3 buckets, Redshift cluster, network IDs) names, as well as internal parameters, like cluster nodes size.
Also, only a public data has been used. Make sure your account version and existing objects' policies fit all your security constraints.
Even though AWS Labs provides us a great repository with all the files necessary: aws-glue-libs, it may not be that easy to set up any version of Glue locally (believe me, I've been there..). Fortunately, there are already Docker images that can be used in that manner. More on that can be found in this AWS Big Data blog post.
Testing using awsgluelibs comes down to run proper Docker commands. It enables to run all of 3 Spark most important commands: spark-submit
, pyspark
, pytest
.
Provided example for spark-submit
can be found below:
WORKSPACE_LOCATION=/local_path_to_workspace
SCRIPT_FILE_NAME=sample.py
mkdir -p ${WORKSPACE_LOCATION}/src
vim ${WORKSPACE_LOCATION}/src/${SCRIPT_FILE_NAME}
docker run -it \
-v ~/.aws:/home/glue_user/.aws \
-v $WORKSPACE_LOCATION:/home/glue_user/workspace/ \
-e AWS_PROFILE=$PROFILE_NAME \
-e DISABLE_SSL=true \
--rm \
-p 4040:4040 \
-p 18080:18080 \
--name glue_spark_submit \
amazon/aws-glue-libs:glue_libs_3.0.0_image_01 \
spark-submit /home/glue_user/workspace/src/$SCRIPT_FILE_NAME
This requires a bit of agility, but is also doable. With the usage of a default Airflow Docker Compose file, it's possible to set up Airflow environment and then modify it to resemble AWS MWAA. The core problem was to be able to assume the same (or at least similar privilege-wise) role as Managed Airflow cluster has. Core problems are connected with IAM, but it's worth researching, because, unfortunately, AWS MWAA DagBag doesn't refresh that often.
The only two command to run are in fact:
docker-compose -f airflow-docker-compose.yaml up airflow-init
docker-compose -f airflow-docker-compose.yaml up
Caution - to smoothly access resources inside VPC (especially Redshift cluster), it's worth adding CIDR block representing local IP / subnet.
- Modify IAM policies in Terraform so roles are not entitled to do anything service-wise (
iam:*
) - it's a huge security leak, but for private one-man-team showcase it was omitted to skip a lot of repeatable Terraform development - Add CI/CD (Github Actions, CircleCI. etc.) to perform Python code formatting (e.g. black)
- Add CI/CD to format Terraform code and apply to the environment according to the branch
- Parametrize all places in code where hard-coded variable values were used (mainly in jobs' scripts, Terraform code, Airflow DAG definition)