carlsvelz / airflow-spark-minio-postgres Goto Github PK

• Table of contents
• Project Requirements
• Project Structure
• High Level Architecture
• 🚢 Containers
• Step-by-Step
  • 1. Clone the Repository
  • 2. Setup environment
  • 3. Airflow: Create user for UI
  • 3.1 Airflow: Postgres, MinIO & Spark connections configuration
    • Postgres
    • MinIO
    • Spark
• Stack
• References

This project was develop and tested using the following environment.

If due to license issues you are not allowed to use Docker it's possible to replace it with Podman which is an alternative highly recommended for companies such as RedHat.

This challenge is currently structured with the following specifications.

• airflow-webserver: Airflow v2.2.4 (Webserver & Scheduler)

  • image: andrejunior/airflow-spark:latest | Based on python:3.8-buster
  • port: 8085

• postgres: Postgres database (Airflow metadata and our pipeline)

  • image: postgres:14-bullseye
  • port: 5432

• spark-master: Spark Master

  • image: bitnami/spark:3.2.1
  • port: 8081

• spark-worker: Spark workers

  • image: bitnami/spark:3.2.1

• MinIO: Object storage

  • image: bitnami/minio:latest

git clone https://github.com/andrejnevesjr/airflow-spark-minio-postgres.git

cd otrium-challenge
docker-compose -f docker-compose.yml up -d

To access Airflow UI is required to create a new user account, so in our case, we are going to create an fictional user with an Admin role attached.

NOTE: Before RUN the command below please confirm that Airflow is up and running, it can be checked by accessing the URL http://localhost:8085. Have in mind that in the first execution it may take 2 to 3 minutes 😜

docker-compose run airflow-webserver airflow users create --role Admin --username airflow
--email [email protected] --firstname airflow --lastname airflow --password airflow

1. Open the service in your browser at http://localhost:8085 Use the credentials

   User: airflow
   Password: airflow
   

2. Click on Admin -> Connections in the top bar.

3. Click on + sign and fill in the necessary details for each source below:

Conn Id: postgres_conn
Conn Type: Postgres
Host: postgres
Schema: airflow
Login: airflow
Password: airflow
Port: 5432

Conn ID: minio_conn
Conn Type: Amazon S3
Extra: consists of the JSON below:

    { "aws_access_key_id":"airflow",
      "aws_secret_access_key": "airflow",
      "host": "http://bucket:9000"
    }

Conn ID: spark_conn
Host: spark://spark
Port: 7077
Extra: consists of the JSON below:

{"queue": "root.default"}

airflow.apache.org

puckel/docker-airflow

cordon-thiago/airflow-spark

pyjaime/docker-airflow-spark