Infrastructure component of a Data Workflow Management system using these components:
- Airflow | version 1.10.10
This repository manages the customised Docker image build of Airflow. The new Docker image is based on Docker Hub apache/airflow. Customised Airflow DAGs and Plugins are built into the image and must be installable as a Python pip
package. This provides an immutable deploy bound within the Docker container during run time. The dependent DAG pip
install defaults to a simple Airflow DAG bookend example.
Note
The dependent DAG pip
install can be overriden to suit your project's requirements as detailed under Image Build.
Get the code and change into the top level git
project directory:
$ git clone https://github.com/loum/data-pipelines-infrastructure.git && cd data-pipelines-infrastructure
Note
Run all commands from the top-level directory of the git
repository.
For first-time setup, get the Makester project:
$ git submodule update --init
Keep Makester project up-to-date with:
$ make submodule-update
Setup the environment:
$ make init
There should be a make
target to be able to get most things done. Check the help for more information:
$ make help
The image build process takes the Docker Hub apache/airflow image and installs a Python package of custom Airflow DAGs and Plugin definitions via the normal Python package management process using pip
. The upstream dependency for the Airflow image build is defined by the DATA_PIPELINES_DAG_REPO
variable in the Makefile
. This defaults to git+https://github.com/loum/[email protected]
which is a simple DAG for the purposes of satisfying the requirement of our new Docker image build. To start the default data-pipelines-infrastructure
Docker image build:
$ make build-image
To list the available data-pipelines-infrastructure
Docker images:
$ make search-image
On the successful build of the Docker image, the typical output would be:
REPOSITORY TAG IMAGE ID CREATED SIZE data-pipelines-infrastructure eda36a5 0671ab41ad37 19 hours ago 766MB
Here, the TAG
is important as it identifies the local data-pipelines-infrastructure
build and is used directly in the Infrastructure Build and Setup.
Note
Triggering the local-build-up
target forces a make build-image
to ensure a data-pipelines-infrastructure
Docker image exists locally.
To build a Dockerised Airflow platform running under Celery Executor mode:
$ make local-build-up
Navigate to the Airflow console http://localhost:<AIRFLOW__WEBSERVER__WEB_SERVER_PORT>
Note
The AIRFLOW__WEBSERVER__WEB_SERVER_PORT
value can be identified with:
make print-AIRFLOW__WEBSERVER__WEB_SERVER_PORT
To release all Docker resources:
$ make local-build-down
To tag the image as latest
:
$ make tag
Or to align with tagging policy <airflow-version>-<data-pipeline-dags-tag>-<image-release-number>
:
$ make tag-version
Note
Control version values by setting MAKESTER__VERSION
and MAKESTER__RELEASE_NUMBER
in the project Makefile.
Kubernetes shakeout and troubleshooting.
- Minikube
- kubectll
- kompose if you would like to convert docker-compose.yml files to Kubernetes manifests
Kubernetes provides the kompose conversion tool that can help you migrate to Kubernetes from docker-compose
. Ensure that your docker-compose.yml
file exists in the top-level directory of your project repository.
To create your Kubernetes manifests:
$ make k8s-manifests
This will deposit the generated Kubernetes manifests under the ./k8s
directory.
Create a Pod and requires Services taken from manifests under ./k8s
directory:
$ make kube-apply
View the Pods and Services:
$ make kube-get
Delete the Pods and Services:
$ make kube-del
The Kubernetes deployment will expose the Airflow Webserver UI that can be browsed to. The URL can be obtained with:
$ minikube service webserver --url
$ make mk-del