This repo tracks details for images necessary to run a proposed MVP Kafka + Cassandra cluster on k8s. It provides automation and documents testing that ensures that containers of these images can run in an air-gapped environment with a basic pub/sub.

(MVP services in bold)

• Apache Cassandra
• Confluent Platform
• Apache Kafka
• Apache NiFi
• Apache Spark
• Dask (Python Parallel Computing)

An expanded manifest with alternate versions also available.

Confluent Platform is the commercial version of Kafka, maintained and sold by Confluent.

Rook is an OSS Kubernetes Operator for Apache Cassandra.
Rook is licensed under the Apache License 2.0.

Strimzi is an OSS Kubernetes Operator for Apache Kafka.
Strimzi is licensed under the Apache License 2.0.

Apache Kafka HQ (AKHQ) is an OSS dashboard webapp for Kafka.
AKHQ is licensed under the Apache License 2.0.

Spark-on-k8s-operator is an unofficial project by Google Cloud Platform for Apache Spark.
Spark-on-k8s-operator is licensed under the Apache License 2.0.

All NiFi related components are licensed under the Apache License 2.0.

Dask is a project by the NumPy, Pandas, Jupyter, Scikit-Learn developer community. It is a community-governed project, and is fiscally sponsored by NumFOCUS - the 501(c)(3) sponsor of NumPy, Pandas, and Jupyter.
Dask is licensed under the BSD 3-Clause License. JupyterHub is licensed under the BSD 3-Clause License.

1. Start Docker for Desktop w/ k8s support enabled
2. Perform the following shell commands:

   # Setup
   export PROJECT=airgapped-kafka-k8s
   git clone --recurse-submodules --shallow-submodules https://github.com/raft-tech/$PROJECT
   cd $PROJECT
   
   # Start local registry and import images
   docker-compose up -d
   [export REGISTRY=<your-custom-registry> && export CP_VERSION=6.0.0 && export CP_OPERATOR_VERSION=6.0.0.0 &&] sh ./scripts/import_images.sh
   
   ## Kafka test
   
   # Confluent Platform Kafka
   # TODO: uses a set of operators from Confluent
   
   # Apache Kafka (FOSS distro)
   [export REGISTRY=<your-custom-registry> && ] sh ./scripts/kafka_local.sh
   
   # Check install
   kubectl get deployments
   helm ls -n operators
   kubectl get deployments -n operators
   kubectl get pods -n operators
   # If jq not installed, either "brew install jq" or remove "| jq" from below
   kubectl get pod -n operators $(kubectl get pods -n operators --no-headers -o custom-columns=":metadata.name") -o jsonpath='{$.spec.containers[*].env[3:]}' | jq
   # Confirm output has image values prepended with "localhost:5000/" 
   # or your custom registry name
   
   # NiFi test
   #[export REGISTRY=<your-custom-registry> && ] sh ./scripts/nfi_local.sh
   #kubectl get pods -n nifi
   
   # Rook test
   [export REGISTRY=<your-custom-registry> && ] sh ./scripts/rook_local.sh
   kubectl get pods -n rook-cassandra-system
   # If jq not installed, either "brew install jq" or remove "| jq" from below
   kubectl get pod -n rook-cassandra-system $(kubectl get pods -n rook-cassandra-system --no-headers -o custom-columns=":metadata.name") -o jsonpath='{$.spec.containers[*].env[3:]}' | jq
   # Confirm output has image values prepended with "localhost:5000/" 
   # or your custom registry name    
   
   # Spark test
   # Install Spark Operator
   helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
   helm install incubator/sparkoperator --generate-name --namespace operators --set sparkJobNamespace=default
   
   # Check install
   helm ls -n operators
   kubectl get deployments -n operators
   kubectl get pods -n operators
   # If jq not installed, either "brew install jq" or remove "| jq" from below
   kubectl get pod -n operators $(kubectl get pods -n operators --no-headers -o custom-columns=":metadata.name") -o jsonpath='{$.spec.containers[*].env[3:]}' | jq
   # Confirm output has image values prepended with "localhost:5000/" 
   # or your custom registry name
   [export REGISTRY=<your-custom-registry> && ] sh ./scripts/spark_local.sh
   kubectl get sparkapplications spark-pi -o=yaml