This project builds my simple structure of data as a service, including some platforms and tools for processing and provide data.
Platform | version |
---|---|
Apache Airflow | 2.1.2 |
Apache Spark | 3.1.2 |
Apache Hadoop | 3.2 |
PostgREST | v7.0.1 |
Need to install | tested version | tested build |
---|---|---|
Docker Engine | 20.10.7 | 20.10.7-0ubuntu1~20.04.2 |
docker-compose | 1.29.2 | 5becea4c |
- First, Build all docker images
make
- Just, run it
make run-all-clusters
Interface | URL |
---|---|
Airflow Webservice | localhost:8080 |
Spark Master Webservice | localhost:8081 |
Spark Worker 1 Webservice | localhost:8082 |
Spark Worker 2 Webservice | localhost:8083 |
Jupyter Lab Webservice | localhost:8888 |
Makefile
- Just, run it
make qas