Infrastructure component of a Data Workflow Management system using these components:

• Airflow | version 1.10.10

This repository manages the customised Docker image build of Airflow. The new Docker image is based on Docker Hub apache/airflow. Customised Airflow DAGs and Plugins are built into the image and must be installable as a Python pip package. This provides an immutable deploy bound within the Docker container during run time. The dependent DAG pip install defaults to a simple Airflow DAG bookend example.

• Docker
• GNU make

Get the code and change into the top level git project directory:

$ git clone https://github.com/loum/data-pipelines-infrastructure.git && cd data-pipelines-infrastructure

For first-time setup, get the Makester project:

$ git submodule update --init

Keep Makester project up-to-date with:

$ make submodule-update

Setup the environment:

$ make init

There should be a make target to be able to get most things done. Check the help for more information:

$ make help

The image build process takes the Docker Hub apache/airflow image and installs a Python package of custom Airflow DAGs and Plugin definitions via the normal Python package management process using pip. The upstream dependency for the Airflow image build is defined by the DATA_PIPELINES_DAG_REPO variable in the Makefile. This defaults to git+https://github.com/loum/[email protected] which is a simple DAG for the purposes of satisfying the requirement of our new Docker image build. To start the default data-pipelines-infrastructure Docker image build:

$ make build-image

To list the available data-pipelines-infrastructure Docker images:

$ make search-image

On the successful build of the Docker image, the typical output would be:

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
data-pipelines-infrastructure   eda36a5             0671ab41ad37        19 hours ago        766MB

Here, the TAG is important as it identifies the local data-pipelines-infrastructure build and is used directly in the Infrastructure Build and Setup.

To build a Dockerised Airflow platform running under Celery Executor mode:

$ make local-build-up

Navigate to the Airflow console http://localhost:<AIRFLOW__WEBSERVER__WEB_SERVER_PORT>

make print-AIRFLOW__WEBSERVER__WEB_SERVER_PORT

To release all Docker resources:

$ make local-build-down

To tag the image as latest:

$ make tag

Or to align with tagging policy <airflow-version>-<data-pipeline-dags-tag>-<image-release-number>:

$ make tag-version

Kubernetes shakeout and troubleshooting.

• Minikube
• kubectll
• kompose if you would like to convert docker-compose.yml files to Kubernetes manifests

Kubernetes provides the kompose conversion tool that can help you migrate to Kubernetes from docker-compose. Ensure that your docker-compose.yml file exists in the top-level directory of your project repository.

To create your Kubernetes manifests:

$ make k8s-manifests

This will deposit the generated Kubernetes manifests under the ./k8s directory.

Create a Pod and requires Services taken from manifests under ./k8s directory:

$ make kube-apply

View the Pods and Services:

$ make kube-get

Delete the Pods and Services:

$ make kube-del

The Kubernetes deployment will expose the Airflow Webserver UI that can be browsed to. The URL can be obtained with:

$ minikube service webserver --url

$ make mk-del