Giter VIP home page Giter VIP logo

apache-nifi-kubernetes's Introduction

Apache NiFi and NiFi Registry on Kubernetes

It is a repository with Helm charts for Apache NiFi and Apache NiFi Registry.

Blog Post about NiFi on Kubernetes.

Note

This repo is constantly being improved.

Prerequisites

You need the following components to use these Helm charts:

  • Kubernetes cluster (1.15 and newer)
  • Helm 3

Tested with the following Ingress:

Tests

You can find CI pipelines for two CICD tools: GitLab CI and GitHub Actions.

  • GitLab CI: .gitlab-ci.yml
  • GitHub Actions: directory .github/workflows

Installation

Create your values file or use the default one.

Apache NiFi:

helm --namespace nifi nifi upgrade --install ./apache-nifi/chart -f ./apache-nifi/chart/values.yaml

Apache NiFi Registry:

helm --namespace nifireg nifi upgrade --install ./apache-nifi-registry/chart -f ./apache-nifi-registry/chart/values.yaml

Configuration - NiFi

The following table lists the configurable parameters of the Apache NiFi chart and the default values.

Parameter Description Default
ReplicaCount
replicaCount Number of nifi nodes 1
Image
image.repository nifi Image name apache/nifi
image.tag nifi Image tag 1.11.4
image.pullPolicy nifi Image pull policy IfNotPresent
image.pullSecret nifi Image pull secret nil
SecurityContext
securityContext.runAsUser nifi Docker User 1000
securityContext.fsGroup nifi Docker Group 1000
sts
sts.podManagementPolicy Parallel podManagementPolicy Parallel
sts.AntiAffinity Affinity for pod assignment soft
sts.pod.annotations Pod template annotations security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000
secrets
secrets Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. nil
configmaps
configmaps Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. nil
nifi properties
properties.externalSecure externalSecure for when inbound SSL false
properties.isNode cluster node properties (only configure for cluster nodes) true
properties.httpPort web properties HTTP port 8080
properties.httpsPort web properties HTTPS port null
properties.clusterPort cluster node port 6007
properties.clusterSecure cluster nodes secure mode false
properties.needClientAuth nifi security client auth false
properties.provenanceStorage nifi provenance repository max storage size 8 GB
properties.siteToSite.secure Site to Site properties Secure mode false
properties.siteToSite.port Site to Site properties Secure port 10000
properties.siteToSite.authorizer managed-authorizer
properties.safetyValve Map of explicit 'property: value' pairs that overwrite other configuration nil
nifi user authentication
auth.ldap.enabled Enable User auth via ldap false
auth.ldap.host ldap hostname ldap://<hostname>:<port>
auth.ldap.searchBase ldap searchBase CN=Users,DC=example,DC=com
auth.ldap.searchFilter ldap searchFilter CN=john
postStart
postStart Include additional libraries in the Nifi containers by using the postStart handler nil
Headless Service
headless.type Type of the headless service for nifi ClusterIP
headless.annotations Headless Service annotations service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
Ingress
ingress.enabled Enables Ingress false
ingress.annotations Ingress annotations {}
ingress.path Path to access frontend (See issue #22) /
ingress.hosts Ingress hosts []
ingress.tls Ingress TLS configuration []
Persistence
persistence.enabled Use persistent volume to store data false
persistence.storageClass Storage class name of PVCs (use the default type if unset) nil
persistence.accessMode ReadWriteOnce or ReadOnly [ReadWriteOnce]
persistence.dataStorage.size Size of persistent volume claim 1Gi
persistence.flowfileRepoStorage.size Size of persistent volume claim 10Gi
persistence.contentRepoStorage.size Size of persistent volume claim 10Gi
persistence.provenanceRepoStorage.size Size of persistent volume claim 10Gi
persistence.logStorage.size Size of persistent volume claim 5Gi
persistence.existingClaim Use an existing PVC to persist data nil
jvmMemory
jvmMemory bootstrap jvm size 2g
SideCar
sidecar.image Separate image for tailing each log separately ez123/alpine-tini
sidecar.tag Image tag latest
BusyBox
busybox.image Separate image for initContainer that verifies zookeeper is accessible busybox
busybox.tag Image tag latest
Resources
resources Pod resource requests and limits for logs {}
logResources
logresources. Pod resource requests and limits {}
nodeSelector
nodeSelector Node labels for pod assignment {}
terminationGracePeriodSeconds
terminationGracePeriodSeconds Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. 60
tolerations
tolerations Tolerations for pod assignment []
initContainers
initContainers Container definition that will be added to the pod as initContainers []
extraVolumes
extraVolumes Additional Volumes available within the pod (see spec for format) []
extraVolumeMounts
extraVolumeMounts VolumeMounts for the nifi-server container (see spec for details) []
env
env Additional environment variables for the nifi-container (see spec for details) []
extraContainers
extraContainers Additional container-specifications that should run within the pod (see spec for details) []

Configuration - NiFi Registry

The following table lists the configurable parameters of the Apache NiFi Registry chart and the default values.

Parameter Description Default
ReplicaCount
replicaCount Number of nifi nodes 1
Image
image.repository nifi Image name apache/nifi
image.tag nifi Image tag 1.11.4
image.pullPolicy nifi Image pull policy IfNotPresent
image.pullSecret nifi Image pull secret nil
SecurityContext
securityContext.runAsUser nifi Docker User 1000
securityContext.fsGroup nifi Docker Group 1000
sts
sts.podManagementPolicy Parallel podManagementPolicy Parallel
sts.AntiAffinity Affinity for pod assignment soft
sts.pod.annotations Pod template annotations security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000
secrets
secrets Pass any secrets to the nifi pods. The secret can also be mounted to a specific path if required. nil
configmaps
configmaps Pass any configmaps to the nifi pods. The configmap can also be mounted to a specific path if required. nil
nifi properties
properties.externalSecure externalSecure for when inbound SSL false
properties.isNode cluster node properties (only configure for cluster nodes) true
properties.httpPort web properties HTTP port 8080
properties.httpsPort web properties HTTPS port null
properties.clusterPort cluster node port 6007
properties.clusterSecure cluster nodes secure mode false
properties.needClientAuth nifi security client auth false
properties.provenanceStorage nifi provenance repository max storage size 8 GB
properties.siteToSite.secure Site to Site properties Secure mode false
properties.siteToSite.port Site to Site properties Secure port 10000
properties.siteToSite.authorizer managed-authorizer
properties.safetyValve Map of explicit 'property: value' pairs that overwrite other configuration nil
nifi user authentication
auth.ldap.enabled Enable User auth via ldap false
auth.ldap.host ldap hostname ldap://<hostname>:<port>
auth.ldap.searchBase ldap searchBase CN=Users,DC=example,DC=com
auth.ldap.searchFilter ldap searchFilter CN=john
postStart
postStart Include additional libraries in the Nifi containers by using the postStart handler nil
Headless Service
headless.type Type of the headless service for nifi ClusterIP
headless.annotations Headless Service annotations service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
Ingress
ingress.enabled Enables Ingress false
ingress.annotations Ingress annotations {}
ingress.path Path to access frontend (See issue #22) /
ingress.hosts Ingress hosts []
ingress.tls Ingress TLS configuration []
Persistence
persistence.enabled Use persistent volume to store data false
persistence.storageClass Storage class name of PVCs (use the default type if unset) nil
persistence.accessMode ReadWriteOnce or ReadOnly [ReadWriteOnce]
persistence.dataStorage.size Size of persistent volume claim 1Gi
persistence.flowfileRepoStorage.size Size of persistent volume claim 10Gi
persistence.contentRepoStorage.size Size of persistent volume claim 10Gi
persistence.provenanceRepoStorage.size Size of persistent volume claim 10Gi
persistence.logStorage.size Size of persistent volume claim 5Gi
persistence.existingClaim Use an existing PVC to persist data nil
jvmMemory
jvmMemory bootstrap jvm size 2g
SideCar
sidecar.image Separate image for tailing each log separately ez123/alpine-tini
sidecar.tag Image tag latest
BusyBox
busybox.image Separate image for initContainer that verifies zookeeper is accessible busybox
busybox.tag Image tag latest
Resources
resources Pod resource requests and limits for logs {}
logResources
logresources. Pod resource requests and limits {}
nodeSelector
nodeSelector Node labels for pod assignment {}
terminationGracePeriodSeconds
terminationGracePeriodSeconds Number of seconds the pod needs to terminate gracefully. For clean scale down of the nifi-cluster the default is set to 60, opposed to k8s-default 30. 60
tolerations
tolerations Tolerations for pod assignment []
initContainers
initContainers Container definition that will be added to the pod as initContainers []
extraVolumes
extraVolumes Additional Volumes available within the pod (see spec for format) []
extraVolumeMounts
extraVolumeMounts VolumeMounts for the nifi-server container (see spec for details) []
env
env Additional environment variables for the nifi-container (see spec for details) []
extraContainers
extraContainers Additional container-specifications that should run within the pod (see spec for details) []

Credits

Inspired from https://github.com/cetic/helm-nifi

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.