The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
Home Page: https://konpyutaika.github.io/nifikop/
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Similar to #79, nifikop might fail to scale down the nificluster if it misses the chance to set gracefulActionState to GracefulUpscaleSucceeded for the nifi node to be deleted.
More concretely, we find that the GracefulActionState.State (in the nificluster cr) for each nifi node (pod) typically goes through the following changes:
GracefulUpscaleRequired inside reconcileNifiPod()GracefulUpscaleRunning inside handlePodAddCCTask()GracefulUpscaleSucceeded inside reconcileNifiPod() when the nifi pod is readyBoth 1 and 3 happens insidereconcileNifiPod(), which is only invoked for each node in the Spec.Nodes as shown below:
for _, node := range r.NifiCluster.Spec.Nodes {
...
o = r.pod(node.Id, nodeConfig, pvcs, log)
err = r.reconcileNifiPod(log, o.(*corev1.Pod))
if err != nil {
return err
}
}
Suppose a user first creates a nificluster with 2 nodes then scales down to 1 node. If the user updates the nificluster cr to remove the last nifi node from Spec.Nodes between step 2 and step 3, GracefulActionState.State of the last nifi pod will never be set to GracefulUpscaleSucceeded and remains as GracefulUpscaleRunning.
Since the GracefulActionState.State of the nifi node is GracefulUpscaleRunning, it will never be added to nodesPendingGracefulDownscale by reconcileNifiPodDelete, and the scale down will never happen.
What did you do?
Scale down nifi cluster
What did you expect to see?
The last nifi pod should be deleted successfully.
What did you see instead? Under which circumstances?
The last nifi pod cannot be deleted and the scale down never happens.
Environment
Possible Solution
Maybe consider invoking reconcileNifiPod() for each currently running nifi pod even if it is not in Spec.Nodes right now
Additional context
We are willing to help fix the bug.
The bug is automatically found by our tool Sieve: https://github.com/sieve-project/sieve
From nifikop created by Piranha900: Orange-OpenSource/nifikop#81
Help around nifikop
What did you do?
i'm trying to set a nifi https cluster with nifikop, i deployed the sample without problems https://github.com/Orange-OpenSource/nifikop/blob/master/config/samples/tls_secured_nificluster.yaml but when i delete the OIDC in overrideConfigs, nifi give me Connection refused on 8443 port.
The zookeeper cluster and cert-manager are deployed as "Get Started" configuration:
https://orange-opensource.github.io/nifikop/docs/2_setup/1_getting_started
and the cluster is exposed like https://orange-opensource.github.io/nifikop/docs/5_references/1_nifi_cluster/7_external_service_config on port 8443.
I tried also to set this configuration https://orange-opensource.github.io/nifikop/docs/3_tasks/2_security/1_ssl but doesn't work ( if you set initialAdminUser you can't deploy because doesn't exist anymore)
Do i need any configuration for certificate authentication?
What did you expect to see?
Nifi asking for authentication
What did you see instead? Under which circumstances?
Connection refused over exposed service.
Environment
nifikop version: 0.5.2
Kubernetes version information: 1.20.4
NiFi version: 1.12.1
What did you do?
namespaces, but ran into permission errors on startup due to missing role bindings.NAMESPACE="nifi"
existing=$(kubectl get namespace ${NAMESPACE})
[ "${existing}" = '' ] && kubectl create namespace ${NAMESPACE}
# You have to create the namespace before executing following command
HELM_EXPERIMENTAL_OCI=1 helm install nifikop \
oci://ghcr.io/konpyutaika/helm-charts/nifikop \
--namespace=${NAMESPACE} \
--version 0.10.0 \
--set image.tag=v0.10.0-release \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set certManager.clusterScoped=true \
--set namespaces={"${NAMESPACE}"}helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nifikop nifi 1 2022-03-27 17:53:38.599003529 +0000 UTC deployed nifikop-0.10.0 0.10.0-release
Applied cert-manager operator using manifests at https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml
Created cloudflare-apitoken-secret using this script
cloudflareSecretName="cloudflare-apitoken-secret";
existingApiToken=$(kubectl -n cert-manager get secret ${cloudflareSecretName} --template='{{.data.apitoken}}' | base64 -d )
if [ "$?" = '0' ]; then
echo "Secret already exists for $cloudflareSecretName with value $existingApiToken";
fi
read -p "Enter the new https://dash.cloudflare.com/profile/api-tokens API Tokens (or nothing to skip/leave as is $existingApiToken): " secret
if [ $secret != "" ]; then
kubectl -n cert-manager delete secret $cloudflareSecretName
kubectl create secret generic -n cert-manager $cloudflareSecretName --from-literal=apitoken=$secret
fi
secret=${secret:-$existingApiToken}
echo "secret is:$secret"
result=$(curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
-H "Authorization: Bearer $secret" \
-H "Content-Type:application/json")
echo $resultapiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# disableAccountKeyGeneration: true
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: [email protected]
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-staging-account-private-key
solvers:
- dns01:
cloudflare:
email: [email protected]
apiTokenSecretRef:
name: cloudflare-apitoken-secret
key: apitoken# The bitnami/zookeeper chart templates create
# fully qualified ZOO_SERVERS hostnames using the
# namespace value.
# eg.
# - name: ZOO_SERVERS
# value: zookeeper-0.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::1 zookeeper-1.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::2 zookeeper-2.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::3
helm template zookeeper bitnami/zookeeper \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set networkPolicy.enabled=true \
--set replicaCount=3 \
--namespace=${NAMESPACE:-tbdNamespace} \
> .target/manifests.yamlapiVersion: nifi.konpyutaika.com/v1alpha1
kind: NifiCluster
metadata:
name: tlsnifi
spec:
service:
headlessEnabled: true
zkAddress: "zookeeper-svc.zookeeper:2181"
zkPath: "/tlsnifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: true
managedAdminUsers:
- identity : "[email protected]"
name: "nifiadmin"
managedReaderUsers:
- identity : "[email protected]"
name: "nifireader"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "2"
memory: 3Gi
requests:
cpu: "1"
memory: 1Gi
nodes:
- id: 0
nodeConfigGroup: "default_group"
- id: 1
nodeConfigGroup: "default_group"
listenersConfig:
clusterDomain: c2.bugslifesolutions.com
useExternalDNS: false
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
sslSecrets:
tlsSecretName: "tlsnifi.c2.bugslifesolutions.com"
create: true
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-staging
externalServices:
- name: "nifi-cluster"
spec:
type: LoadBalancer
portConfigs:
- port: 8443
internalListenerName: "https"
readOnlyConfig:
# NifiProperties configuration that will be applied to the node.
nifiProperties:
webProxyHosts:
- nifi.cluster2.admin.bugslifesolutions.com
- nifi.c2.admin.bugslifesolutions.com
# Additionnals nifi.properties configuration that will override the one produced based
# on template and configurations.
overrideConfigs: |
nifi.ui.banner.text=NiFi
nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration
nifi.security.user.oidc.client.id=930711295780-i72an91pqj7rib88r23qfv5q7mth8hgv.apps.googleusercontent.com
nifi.security.user.oidc.client.secret=BvmEyr81P0YXZtIt1FIfGsRs
nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
What did you expect to see?
A pod created for the NifiCluster.
What did you see instead? Under which circumstances?
No pods were created for the NifiCluster and the status.state is ClusterReconciling:
status:
nodesState:
"0":
configurationState: ""
gracefulActionState:
actionState: ""
errorMessage: ""
initClusterNode: true
podIsReady: false
prometheusReportingTask:
id: ""
version: 0
rollingUpgradeStatus:
errorCount: 0
lastSuccess: ""
state: ClusterReconciling
The Nifikop operator logs suggest the secrets aren't created appropriately:
{"level":"info","ts":1648403623.7630396,"logger":"setup","msg":"manager set up with multiple namespaces","namespaces":"nifi"}
...
{"level":"info","ts":1648403625.282073,"logger":"controller.nifidataflow","msg":"Starting workers","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiDataflow","worker count":1}
{"level":"info","ts":1648403877.1215262,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterInitializing"}
{"level":"info","ts":1648403877.1492836,"logger":"controllers.NifiCluster","msg":"Nifi cluster state updated"}
{"level":"info","ts":1648403877.1786754,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterInitialized"}
{"level":"info","ts":1648403877.2189906,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterReconciling"}
{"level":"info","ts":1648403877.2191544,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi"}
{"level":"info","ts":1648403877.3570118,"logger":"controllers.NifiCluster","msg":"resource created","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi","kind":"*v1.Service","name":"tlsnifi-headless"}
{"level":"info","ts":1648403877.3572803,"logger":"controllers.NifiCluster","msg":"A new resource was not found or may not be ready"}
{"level":"info","ts":1648403877.3573396,"logger":"controllers.NifiCluster","msg":"server secret not ready: Secret \"tlsnifi-0-server-certificate\" not found"}
{"level":"info","ts":1648403877.3895488,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterReconciling"}
{"level":"info","ts":1648403877.3896518,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi"}
{"level":"info","ts":1648403877.391266,"logger":"controllers.NifiCluster","msg":"resource updated","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi","kind":"*v1.Service","name":"tlsnifi-headless"}
{"level":"info","ts":1648403877.3913736,"logger":"controllers.NifiCluster","msg":"A new resource was not found or may not be ready"}
{"level":"info","ts":1648403877.3913915,"logger":"controllers.NifiCluster","msg":"server secret not ready: Secret \"tlsnifi-0-server-certificate\" not found"}
{"level":"info","ts":1648403877.6715496,"logger":"controllers.NifiUser","msg":"generated secret not found, may not be ready"}
{"level":"info","ts":1648403877.796963,"logger":"controllers.NifiUser","msg":"failed to reconcile user secret"}
{"level":"error","ts":1648403877.7970812,"logger":"controller.nifiuser","msg":"Reconciler error","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiUser","name":"tlsnifi-0-node.tlsnifi-headless.nifi.svc.c2.bugslifesolutions.com","namespace":"nifi","error":"could not create user certificate: admission webhook \"webhook.cert-manager.io\" denied the request: spec.commonName: Too long: must have at most 64 bytes","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1648403877.7981198,"logger":"controllers.NifiUser","msg":"generated secret not found, may not be ready"}
{"level":"info","ts":1648403877.8216078,"logger":"controllers.NifiUser","msg":"failed to reconcile user secret"}
{"level":"error","ts":1648403877.8217096,"logger":"controller.nifiuser","msg":"Reconciler error","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiUser","name":"tlsnifi-0-node.tlsnifi-headless.nifi.svc.c2.bugslifesolutions.com","namespace":"nifi","error":"could not create secret with jks password: secrets \"tlsnifi-0-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
The Cert-Manager pod logs indicate an issue with the Domain of the Nifikop tlsnifi-controller owned CertificateRequest:
E0327 18:10:13.819600 1 sync.go:270] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"tlsnifi-controller\": Domain name needs at least one dot" "resource_kind"="Order" "resource_name"="tlsnifi-controller-hjpcs-2838947340" "resource_namespace"="nifi" "resource_version"="v1"
in context:
I0327 18:09:00.095912 1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-account-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0327 18:09:05.034646 1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-account-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0327 18:10:03.378491 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Ready" to 2022-03-27 18:10:03.37839815 +0000 UTC m=+63.713073736
I0327 18:10:03.378861 1 trigger_controller.go:181] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="nifi/tlsnifi-controller" "message"="Issuing certificate as Secret does not contain a private key" "reason"="MissingData"
I0327 18:10:03.378913 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Issuing" to 2022-03-27 18:10:03.378908921 +0000 UTC m=+63.713584482
I0327 18:10:03.762733 1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi-controller" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi-controller\": the object has been modified; please apply your changes to the latest version and try again"
I0327 18:10:03.762897 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Ready" to 2022-03-27 18:10:03.762888653 +0000 UTC m=+64.097564227
I0327 18:10:12.499043 1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi-controller" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi-controller\": the object has been modified; please apply your changes to the latest version and try again"
I0327 18:10:12.591206 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi-controller-hjpcs" condition "Approved" to 2022-03-27 18:10:12.591190353 +0000 UTC m=+72.925865914
I0327 18:10:12.670590 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi-controller-hjpcs" condition "Ready" to 2022-03-27 18:10:12.670574343 +0000 UTC m=+73.005249913
E0327 18:10:13.819600 1 sync.go:270] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"tlsnifi-controller\": Domain name needs at least one dot" "resource_kind"="Order" "resource_name"="tlsnifi-controller-hjpcs-2838947340" "resource_namespace"="nifi" "resource_version"="v1"
I0327 18:10:13.938765 1 conditions.go:190] Found status change for Certificate "tlsnifi-controller" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2022-03-27 18:10:13.938746383 +0000 UTC m=+74.273421949
I0327 18:10:13.965550 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="nifi/tlsnifi-controller" "retry_delay"=3599034548569
I0327 18:10:14.023466 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="nifi/tlsnifi-controller" "retry_delay"=3598976585445
Here is the CertificateRequest:
apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
annotations:
cert-manager.io/certificate-name: tlsnifi-controller
cert-manager.io/certificate-revision: "1"
cert-manager.io/private-key-secret-name: tlsnifi-controller-s5j8v
creationTimestamp: "2022-03-27T18:10:12Z"
generateName: tlsnifi-controller-
generation: 1
name: tlsnifi-controller-hjpcs
namespace: nifi
ownerReferences:
- apiVersion: cert-manager.io/v1
blockOwnerDeletion: true
controller: true
kind: Certificate
name: tlsnifi-controller
uid: 83fecd4e-6f95-45f8-9d9c-612186ce9eb1
resourceVersion: "23860749"
uid: c7100e70-6c6e-404a-a9b6-3839d5227795
spec:
extra:
authentication.kubernetes.io/pod-name:
- cert-manager-786b9c87f8-nsb8z
authentication.kubernetes.io/pod-uid:
- 035536c1-97a2-49c3-84a8-f64e76c23e98
groups:
- system:serviceaccounts
- system:serviceaccounts:cert-manager
- system:authenticated
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-staging
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRStEQ0NBdUFDQVFBd0hURWJNQmtHQTFVRUF4TVNkR3h6Ym1sbWFTMWpiMjUwY205c2JHVnlNSUlDSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUF2bWlXLzZvVTNSMmVUK080M25HdmkwT3BoYUdICldHSUNreU5hT0UxNVEwV3dsSmF4dXFpMVZXcG9RTGVBL2liQUpqaEpJWlFMSGErUkpnN3piWWdYUFB2cnkzcG0KRmhjcC94NG95WVBNSFhIdytseFdzYlMvb2l0a2g5OWJMWFA1aWNFRTlGMW9nQWpTNTFCWXpUdDZNWDFYQmdSYwpkZ0RRMUVTdFFaS3p3WTF6UERqSHUwTW90eUh4RXg4bVlmNlUwY1pQa0owYkpDUkdob1dZVUZ5alFMMmROK1QxClY1dDdBdG9zSUhRYW1sTHIxWXZmZy9OYVljRHNOWjNIaWVUTTZNM1NsWWZ3TlE5eTFzMTRWVks2Nzl4bEJJUFcKd0xHYWo5SmVmZWZsa0V0MXQ3a2xqK2kvRUZGb05xcERQOXVFajhFTVJsNjA5S2xRYzkzUC9vaTdqUi9TdXFZdwp1Q1NWeVJBVElSMjd5eSsvU0VSUnRuYnc0Y3NHbkJ0SFA4czVlMTdxVGdkak02QjZLblFTWSs4NEhtUFh1cGRGCjBuZDhWUXFsZTl6a2hYVzh1VTN2K1k3RXZUdytUTVVLTDFjZTBsdDNxaG45ZkpsYUsweE8yT2w3UFk2N2xjcGMKdG9KdlBDSXhBV2doMXEwQkE4ZzJyV0gzRGV3VzhuZDYvMmNyN241ZXA3ZUl5aUlLSHFBb3VkUDBENDJ3UlRSUwpsaFYyUlV1VnZ4aUplUmdtbWp2VCtXWi9PWnRUa2ZNUkJCVjNtYUs4VUNOYzBUejhReWhzdWpnejIzcU5vakt4CjBZNHgvWFlhYVE1MitZY1ZUUE9IWHdCdmwwWUVzb0h4M08ydDBuaWI4Tk5BelRVbzIwWHgweXhDOC9yZ3gzSEcKVDdsaHRpcExpUUFBc3JVQ0F3RUFBYUNCbFRDQmtnWUpLb1pJaHZjTkFRa09NWUdFTUlHQk1GTUdBMVVkRVFSTQpNRXFDRW5Sc2MyNXBabWt0WTI5dWRISnZiR3hsY29ZMGMzQnBabVpsT2k4dmRHeHpibWxtYVM5dWN5OXVhV1pwCkwyNXBabWwxYzJWeUwzUnNjMjVwWm1rdFkyOXVkSEp2Ykd4bGNqQUxCZ05WSFE4RUJBTUNBQUF3SFFZRFZSMGwKQkJZd0ZBWUlLd1lCQlFVSEF3SUdDQ3NHQVFVRkJ3TUJNQTBHQ1NxR1NJYjNEUUVCRFFVQUE0SUNBUUNxUGxBTApNaGRSZzBRaVFtNzQwKy9GQXJicjRqUjVoSWZXelhXN1pON0xqbXBmMkJSZlpnbUNHOFdRQVMzWWtQeXFicVdOCitubTcwYWRZdVAweHNFTHEwRi9aL096R05kU0J0OEdVSHJ2di9NalYzVUl4bjVKUDgyYTBVcXpibXU5VVhOMHgKMXFnOElQcDVuMTBoSUJmWDViK1Jabm9pYnowcmtjcVNqMjUyTnZZd3lkNmt6cTV2ejRnQ2hjVW9CZEhOeXBkMgphL0dxQWkrYm1RUSswanJyejJ6dUppNC9JUlF5eTNCT1FSRW5tVDJCMlh5YVhtQnJtZ1hmZytTbGdzVWN0NEFhClE2MXBFQUVKT3F3aWtVbmZXbndqcnh5Y0xoYkR0aGo0R0drZUgzdmplbWtkbENBMWdxSTJGVmU5WWJpWkt6KzIKNlg4bGxRNlAzaDhyY1pNbVRUcUl1eldIMFEzZVBUWTdtZGt1YW1hR1h2TisvM3hkTlNiaFZuckwycHNyTFFXUwpxM096aXB1WFM5UFAySDk2cVplSy8wWlpDWFhpNXc4czlPZW1RdUtFcXF3RkJIYXdSckY1SE5UTWk1ekVuTjZpCm1xZ2trNmdZZmk0L2RFRFdSeEJ6enFxMHdPOThRUkxsUUkxOVQ5R1JzMk1WaXpBM0pNUUNlcHRnVlR3R3lDUnkKN3pqNkQwUEZ5Z0gvMmt5RGlsZ0NGcUVSYkd1K2R3RmhmejhFNUIwVElmVC9DSXlhYWtmMFA4N3VQVnp4N0hEVwpCaDZEckxmbWRlRDNjR1htL2tPT09NT21OaWkyZ0FFVzh3RTBkQWdRam9BbkFSUE1VMDNIcE5QeFJhQXVmZXFTCmJ4cm5CdmZtSHhrVnlwd01abjJDYTJQVEZvbmxOREozZk1hYmtBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
uid: 62f7108a-f2ba-4b63-b558-e52ec01a78aa
usages:
- client auth
- server auth
username: system:serviceaccount:cert-manager:cert-manager
status:
conditions:
- lastTransitionTime: "2022-03-27T18:10:12Z"
message: Certificate request has been approved by cert-manager.io
reason: cert-manager.io
status: "True"
type: Approved
- lastTransitionTime: "2022-03-27T18:10:12Z"
message: 'Failed to wait for order resource "tlsnifi-controller-hjpcs-2838947340"
to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:rejectedIdentifier:
Error creating new order :: Cannot issue for "tlsnifi-controller": Domain name
needs at least one dot'
reason: Failed
status: "False"
type: Ready
failureTime: "2022-03-27T18:10:13Z"
Environment
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: privileged
creationTimestamp: "2022-03-27T17:53:40Z"
generateName: nifikop-5d7d6fb5cd-
labels:
app: nifikop
control-plane: nifikop
name: nifikop
operator: nifi
pod-template-hash: 5d7d6fb5cd
release: nifikop
name: nifikop-5d7d6fb5cd-trjll
namespace: nifi
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: nifikop-5d7d6fb5cd
uid: 4e13caad-5cae-47f4-82fc-93b1b391beec
resourceVersion: "23854732"
uid: 8b09a927-397d-432b-a6d5-7a0264e6324e
spec:
containers:
- args:
- metrics-bind-address=8081
- --leader-elect
- --cert-manager-enabled=true
command:
- /manager
env:
- name: WATCH_NAMESPACE
value: nifi
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: OPERATOR_NAME
value: nifikop
- name: LOG_LEVEL
value: Info
image: ghcr.io/konpyutaika/docker-images/nifikop:v0.10.0-release
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8081
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 1
name: nifikop
ports:
- containerPort: 8081
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: 8081
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 250m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-rh9rl
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: worker5
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsUser: 1000
serviceAccount: nifikop
serviceAccountName: nifikop
terminationGracePeriodSeconds: 10
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-rh9rl
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:40Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:50Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:50Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:40Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://e9836ba7f894653319d3d8fed0549270b6323083f04330b54b909cb775a7b782
image: ghcr.io/konpyutaika/docker-images/nifikop:v0.10.0-release
imageID: ghcr.io/konpyutaika/docker-images/nifikop@sha256:99a59041be7cf06e69f4112b9e1e2c6ee6a0320a7942680bc7daf1d13da0a6be
lastState: {}
name: nifikop
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2022-03-27T17:53:43Z"
hostIP: 10.5.2.61
phase: Running
podIP: 10.244.12.80
podIPs:
- ip: 10.244.12.80
qosClass: Guaranteed
startTime: "2022-03-27T17:53:40Z"Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Kubernetes cluster kind:
NiFi version:
apache/nifi:1.12.1
Possible Solution
Suffix the auto-created controller Common Name/Domain with the NifiCluster's spec.listenersConfig.clusterDomain
eg. tlsnifi-controller.c2.bugslifesolutions.com
Pattern: <metadata.name>-controller.<listenersConfig.clusterDomain>
Additional context
From nifikop created by teplydat: Orange-OpenSource/nifikop#66
What did you do?
I want to grant permission on specific "components" (let's call it "itam". So it is a process group under "NiFi Flow") via CR "NifiUserGroup".
Doc is here: https://orange-opensource.github.io/nifikop/docs/v0.4.3/3_tasks/4_nifi_user_group
...
- type: component
action: read
resource: /
componentType: process-groups
componentId: 1ac3ab15-0177-1000-0000-000017feb4b2
# componentId: "1ac3ab15-0177-1000-0000-000017feb4b2"
# componentId: "itam"
I tried different things for componentId.
What did you expect to see?
User policy should be for example:
"Component policy for process Group itam"
What did you see instead? Under which circumstances?
No such policy.
If I don't specify "componentId" then it is:
"Component policy for process Group NiFi Flow"
Environment
nifikop version: nifikop: v0.4.2-alpha-release
go version: -
Kubernetes version information: OpenShift 4.6.z, K8s 1.19
Kubernetes cluster kind: ?
NiFi version: nifi-1.11.4-RC1
Possible Solution
It seems the componentId is not set here: https://github.com/Orange-OpenSource/nifikop/blob/08e3fc742d96d630287b244f95b4e2dbeb2890a1/pkg/clientwrappers/accesspolicies/policies.go#L132
Would be very nice if someone can help!
The deployment of the example command line breaks.
kubectl create -n nifi -f config/samples/simplenificluster.yaml
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.7", GitCommit:"b56e432f2191419647a6a13b9f5867801850f969", GitTreeState:"clean", BuildDate:"2022-02-16T11:50:27Z", GoVersion:"go1.16.14", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10-gke.2000", GitCommit:"0823380786b063c3f71d5e7c76826a972e30550d", GitTreeState:"clean", BuildDate:"2022-03-17T09:22:22Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}
➜ nifikop git:(master) kubectl create -n nifi -f config/samples/simplenificluster.yaml I0505 21:55:30.757329 63607 versioner.go:58] no Auth Provider found for name "gcp" error: error validating "config/samples/simplenificluster.yaml": error validating data: [ValidationError(NifiCluster.spec.externalServices[1]): unknown field "serviceAnnotations" in com.konpyutaika.nifi.v1alpha1.NifiCluster.spec.externalServices, ValidationError(NifiCluster.spec.externalServices[2]): unknown field "serviceAnnotations" in com.konpyutaika.nifi.v1alpha1.NifiCluster.spec.externalServices]; if you choose to ignore these errors, turn validation off with --validate=false
To solve I need to remove the serviceAnnotations but it is strange. I think the problem is elsewhere because it runs the creation but no way to use Nifi after.
Thanks a lot,
Best regards
From nifikop created by erdrix: Orange-OpenSource/nifikop#115
Is your feature request related to a problem? Please describe.
There is no way to automate NiFi datafllow validation test.
Describe the solution you'd like to see
Define a resource like NifiDataflowFunctionalTest to define a list of test to play.
Something like :
nifiDataflowTest:
nifiDataflowRef: NifiDataflowSpec
inputsData:
- content: string
attributes: map[string]string
injectComponentRef: string
checkAssertions:
- connectionRef: string
content:
kind: [exactlyMatch | regexMatch]
value: string
attributes: map[string]{kind: [exactlyMatch | regexMatch], value: string}
disableComponentRefs: list(string)With a logic :
nifiDataflowTest.nifiDataflowRef ,nifiDataflowTest.disableComponentRefs ,nifiDataflowTest.checkAssertions[*].connectionRef ,GenerateFlowfile for each elements of nifiDataflowTest.inputsData and create a connection to the component referenced in nifiDataflowTest.inputsData[*].injectComponentRefnifiDataflowTest.checkAssertions[*].connectionRef,if the connection contains an element, if so, compare the content and attributes of the flowfile with the associated. assertion If it doesn't match = Test failed, if it does match, start the output component of the connection. And this until one assertion fails or all assertions haveFrom nifikop created by andrew-musoke: Orange-OpenSource/nifikop#201
Are you asking about community best practices, how to implement a specific feature, or about general context and help around nifikop ?
General help with Nifikop.
What did you do?
I deployed Nifi with 2 pods via NifiKops. After creating a flow on the UI, I exported the process groups to a nifi-registry as well. The cluster run for days. This is the CR I used. I then deleted the cluster pods to test resilience.
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: simplenifi
namespace: dataops
spec:
service:
headlessEnabled: true
zkAddress: "zookeeper.dataops.svc.cluster.local.:2181"
zkPath: "/simplenifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: false
nodeConfigGroups:
default_group:
isNode: true
imagePullPolicy: IfNotPresent
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "0.5"
memory: 2Gi
requests:
cpu: "0.5"
memory: 2Gi
clientType: "basic"
nodes:
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
listenersConfig:
internalListeners:
- type: "http"
name: "http"
containerPort: 8080
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000What did you expect to see?
I expected the cluster to run properly and survive restarts since PVs are created. I expected to see the pipelines continue running after the pods started up.
What did you see instead? Under which circumstances?
When the pods came back up and were healthy, the UI had no flows or process groups. The registry configuration had also disappeared. I have to manually re-register the nifi-registry, re-import the process groups, add the secrets and restart the pipelines.
Environment
nifikop version:
v0.7.5-release
Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-eks-f17b81", GitCommit:"f17b810c9e5a82200d28b6210b458497ddfcf31b", GitTreeState:"clean", BuildDate:"2021-10-15T21:46:21Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
apache/nifi:1.12.1
From nifikop created by iordaniordanov: Orange-OpenSource/nifikop#186
What did you do?
Deploy nifi secure nifi cluster without Cert-Manager deployed in my K8S cluster. Yaml used to test
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: <name>
labels:
app: <label>
spec:
service:
headlessEnabled: true
zkAddress: <zk_address>
zkPath: <zk_path>
clusterImage: <Nifi Image>
oneNifiNodePerNode: true
clusterSecure: true
readOnlyConfig:
logbackConfig:
replaceConfigMap:
data: logback.xml
name: <logback configmap name>
namespace: <logback configmap namespace>
bootstrapProperties:
nifiJvmMemory: <Memory>
nifiProperties:
webProxyHosts:
- "<proxy_host>"
overrideConfigs: |
nifi.nar.library.autoload.directory=./extensions
nifi.web.http.network.interface.default=eth0
nifi.web.http.network.interface.lo=lo
nifi.web.proxy.context.path=<context_path>
nifi.database.directory=/opt/nifi/data/database_repository
nifi.flow.configuration.archive.dir=/opt/nifi/data/archive
nifi.flow.configuration.file=/opt/nifi/data/flow.xml.gz
nifi.templates.directory=/opt/nifi/data/templates
nifi.provenance.repository.max.storage.size=<max_size>GB
nifi.provenance.repository.indexed.attributes=<attributes>
nodeConfigGroups:
default_group:
isNode: true
nodeSelector:
<node_selectors>
storageConfigs:
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <size>Gi
storageClassName: <storage_class>
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <size>Gi
storageClassName: <storage_class>
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <size>Gi
storageClassName: <storage_class>
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <size>Gi
storageClassName: <storage_class>
- mountPath: "/opt/nifi/nifi-current/work"
name: work
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: <size>Gi
storageClassName: <storage_class>
serviceAccountName: "default"
resourcesRequirements:
<resource_requirements>
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
listenersConfig:
internalListeners:
- type: "http"
name: "http"
containerPort: <port>
- type: "cluster"
name: "cluster"
containerPort: <port>
- type: "s2s"
name: "s2s"
containerPort: <port>
- type: "prometheus"
name: "prometheus"
containerPort: <port>
sslSecrets:
tlsSecretName: "test-nifikop"
create: false
externalServices:
- name: "clusterip"
spec:
type: ClusterIP
portConfigs:
- port: <port>
internalListenerName: "http"
The same YAML works for our insecure Nifi deployments just by removing
sslSecrets:
tlsSecretName: "test-nifikop"
create: false
and setting
clusterSecure: false
What did you expect to see?
I'm sure there are issues with our config and I'm not expecting it to work from the first time, but I was expecting at least to see pods being brought up and having some kind of errors in the Nifi logs.
What did you see instead? Under which circumstances?
No pods are being brought up and I see errors in the Nifi Operator
{"level":"error","ts":1643719642.1879852,"logger":"controller-runtime.manager.controller.nifiparametercontext","msg":"Reconciler error","reconciler group":"nifi.orange.com","reconciler kind":"NifiParameterContext","name":"<parameter_context_name>","namespace":"<namespace>","error":"controller secret not found: Secret \"<cluster_name>-controller\" not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:267\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
{"level":"info","ts":1643719644.999143,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"<cluster_name>","clusterNamespace":"<namespace>"}
{"level":"error","ts":1643719647.3088443,"logger":"controller-runtime.manager.controller.nifiparametercontext","msg":"Reconciler error","reconciler group":"nifi.orange.com","reconciler kind":"NifiParameterContext","name":"<parameter_context_name>","namespace":"<namespace>","error":"controller secret not found: Secret \"<cluster_name>-controller\" not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:267\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}
{"level":"info","ts":1643719652.2984219,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"<cluster_name>","clusterNamespace":"<namespace>"}
From what I understand the operator is looking for a secret named <cluster_name>-controller but I haven't referenced it anywhere I don't understand why it is looking for it
Environment
0.7.1
btw, you should implemet --version command
Which ever version is included in your docker image - https://hub.docker.com/layers/orangeopensource/nifikop/v0.7.1-release/images/sha256-94eb3f1ec21f52041cc832fe9c1124cd6ad87666af8ffb41cd9173ae97f9a0c3?context=explore
v1.19.15-eks-9c63c4
Kubernetes cluster kind: EKS
NiFi version:
1.15.3
Possible Solution
Additional context
Add any other context about the problem here.
From nifikop created by riccardo-salamanna: Orange-OpenSource/nifikop#129
What did you do?
Trying to scale up and scale down a cluster
What did you expect to see?
The cluster scaling up and caling down when i add nodes
What did you see instead? Under which circumstances?
The scaling down does not happen, only the scaling up (and also every other configuration change does trigger a refresh). The log of the operator are also filled with errors and it's CPU usage does spike and stay high.
Environment
nifikop version: 0.6.3-release
go version:
Kubernetes version information:1.19
Kubernetes cluster kind: EKS
NiFi version: tried multiple, 1.11.3, 1.12.1, 1.13.2
Possible Solution
I sincerely do not know.
Additional context
here's the output log for the operator pod
2021-09-02T17:21:20.541Z ERROR nifi_client Error during preparing the request {"error": "The target node id doesn't exist in the cluster", "errorVerbose": "The target node id doesn't exist in the cluster\ngithub.com/Orange-OpenSource/nifikop/pkg/nificlient.init\n\t/workspace/pkg/nificlient/common.go:27\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5652\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5647\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5647\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:5647\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:191\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/Orange-OpenSource/nifikop/pkg/nificlient.(*nifiClient).GetClusterNode
/workspace/pkg/nificlient/system.go:49
github.com/Orange-OpenSource/nifikop/pkg/clientwrappers/scale.CheckIfNCActionStepFinished
/workspace/pkg/clientwrappers/scale/scale.go:166
github.com/Orange-OpenSource/nifikop/controllers.(*NifiClusterTaskReconciler).checkNCActionStep
/workspace/controllers/nificlustertask_controller.go:324
github.com/Orange-OpenSource/nifikop/controllers.(*NifiClusterTaskReconciler).handlePodRunningTask
/workspace/controllers/nificlustertask_controller.go:251
github.com/Orange-OpenSource/nifikop/controllers.(*NifiClusterTaskReconciler).Reconcile
/workspace/controllers/nificlustertask_controller.go:89
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99
From nifikop created by teplydat: Orange-OpenSource/nifikop#58
I got the error message:
"error":"could not create user certificate: admission webhook \"webhook.cert-manager.io\" denied the request: spec.commonName: Too long: must have at most 64 bytes"
I think you are aware of this possible problem (mentioned in Orange-OpenSource/nifikop#21).
I think its not a good idea to allow more than 64 bytes for the CN or DNS because of RFC standards.
If I have a look on the CN: "nifi-0-node.nifi-headless.name-space-longername.svc.cluster.local" (65 bytes)
I can separate it in the following pieces:
nifi (name already quite short)
-0-node (to have a unique name for the nifi nodes)
nifi-headless (quite long)
namespace-longer-name (=namespace)
.svc.cluster.local (k8s specific. probably not changable)
What is your plan about that topic? Can we shorten something?
From nifikop created by teplydat: Orange-OpenSource/nifikop#39
Are you asking about community best practices, how to implement a specific feature, or about general context and help around nifikop ?
Best practice how to mount an existing pvc on nifi
What did you do?
At first I created a pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fstp-pvc
namespace: usecase
labels:
pvc: fstp
spec:
storageClassName: "ceph-fs-storage"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
Then I tried to mount it via labels though the nificlusters.nifi.orange.com:
...
storageConfigs:
- mountPath: "/opt/fstp"
name: fstp-pvc
pvcSpec:
accessModes:
- ReadWriteMany
selector:
matchLabels:
pvc: fstp
...
What did you expect to see?
Nifi mounts the existing pvc.
What did you see instead? Under which circumstances?
No nifi node is scheduled by the operator.
logs from the operator:
PersistentVolumeClaim \"nifi-0-storagebb7tt\" is invalid: spec.resources[storage]: Required value","Request.Namespace":"usecase","Request.Name":"nifi"}
{"level":"error","ts":1603277145.6576192,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"nificluster-controller","request":"usecase/nifi","error":"failed to reconcile resource: creating resource failed: PersistentVolumeClaim \"nifi-0-storagebb7tt\" is invalid: spec.resources[storage]: Required value","errorVerbose":"creating resource failed: PersistentVolumeClaim \"nifi-0-storagebb7tt\" is invalid: spec.resources[storage]: Required value\nfailed to reconcile
Environment
image: orangeopensource/nifikop:v0.2.0-release
v1.16.7
nificlusters.nifi.orange.com
1.11.4
From nifikop created by wandersonpereira: Orange-OpenSource/nifikop#178
Hi Guys!
I'm trying to start nifi in k8s and I've configured sslSecrets, but it's responsing error of decode PEM.
2021-12-15T14:05:56.529Z ERROR controller-runtime.manager.controller.nificluster Reconciler error {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "name": "nifi-cluster", "namespace": "operator", "error": "failed to decode certificate: Failed to decode x509 certificate from PEM", "errorVerbose": "Failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(*Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:441\ngithub.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:148\ngithub.com/Orange-OpenSource/nifikop/controllers.(*NifiClusterReconciler).Reconcile\n\t/workspace/controllers/nificluster_controller.go:135\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"}
I've tried same configuration of the tutorials, but it not found.
My Configuration:
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: nifi-cluster
spec:
service:
headlessEnabled: true
zkAddress: "MY_SERVICE_OF_ZK:2181"
zkPath: "/hmlnificluster"
clusterImage: "apache/nifi:1.13.2"
oneNifiNodePerNode: false
managedAdminUsers:
- identity : "[email protected]"
name: "wanderson.pereira"
managedReaderUsers:
- identity : "[email protected]"
name: "wanderson.pereira"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
nifiProperties:
webProxyHosts:
- nifi-hubb2b-hml.ifcshop.com.br:8443
overrideConfigs: |
nifi.web.http.network.interface.default=eth0
nifi.web.http.network.interface.lo=lo
xxxx
bootstrapNotificationServicesConfig:
replaceConfigMap:
name: config-map-nifi
namespace: operator
data: bootstrap_notifications_services.xml
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-storage"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "1.25"
memory: 4Gi
requests:
cpu: "1.25"
memory: 2Gi
nodes:
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
listenersConfig:
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
- type: "prometheus"
name: "prometheus"
containerPort: 9090
sslSecrets:
tlsSecretName: "nifi-staging-tls"
create: true
I tried to use same version of cert-manager:
Environment
nifikop version: 0.7.5
Kubernetes version information: 1.20.9
Kubernetes cluster kind: AKS
NiFi version: apache/nifi:1.13.2
From nifikop created by Docteur-RS: Orange-OpenSource/nifikop#166
Requesting help on strange behavior
Hey,
Followed the installation tutorial from the documentation but after the last step nothing happend.
What I did :
# Installing zookeeper (in default namespace...)
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install zookeeper bitnami/zookeeper \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set global.storageClass=standard \
--set networkPolicy.enabled=true \
--set replicaCount=3
# Installing certmanager
kubectl apply -f \
https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml
# Installing nifikops
helm repo add orange-incubator https://orange-kubernetes-charts-incubator.storage.googleapis.com/
kubectl create ns nifi
helm install nifikop \
orange-incubator/nifikop \
--namespace=nifi \
--version 0.7.5 \
--set image.tag=v0.7.5-release \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set namespaces={"nifi"}
Next, I didn't setup the customstorage class because it was optional.
# Applying the simple nifi cluster
## Updated the storage StorageClassName to the one availble inside my cluster
kubectl create -n nifi -f config/samples/simplenificluster.yaml
And now..... Nothing.
kubectl get po -n default
NAME READY STATUS RESTARTS AGE
zookeeper-0 1/1 Running 0 40m
zookeeper-1 1/1 Running 0 40m
zookeeper-2 1/1 Running 0 40m
kubectl get po -n nifi
NAME READY STATUS RESTARTS AGE
nifikop-5d4c9b6d6d-mz922 1/1 Running 0 33m
kubectl get nificluster
NAME AGE
simplenifi 32m
#Nifikop logs
2021-11-29T17:27:07.597Z INFO setup manager set up with multiple namespaces {"namespaces": "nifi"}
2021-11-29T17:27:07.597Z INFO setup Writing ready file.
I1129 17:27:08.718212 1 request.go:655] Throttling request took 1.013748334s, request: GET:https://10.3.0.1:443/apis/apm.k8s.elastic.co/v1?timeout=32s
2021-11-29T17:27:09.984Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2021-11-29T17:27:09.991Z INFO setup starting manager
2021-11-29T17:27:09.992Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-11-29T17:27:09.993Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:09.993Z INFO controller-runtime.manager.controller.nifiuser Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUser", "source": "kind source: /, Kind="}
2021-11-29T17:27:09.994Z INFO controller-runtime.manager.controller.nifiusergroup Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUserGroup", "source": "kind source: /, Kind="}
2021-11-29T17:27:09.994Z INFO controller-runtime.manager.controller.nifidataflow Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiDataflow", "source": "kind source: /, Kind="}
2021-11-29T17:27:09.994Z INFO controller-runtime.manager.controller.nifiparametercontext Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiParameterContext", "source": "kind source: /, Kind="}
2021-11-29T17:27:09.994Z INFO controller-runtime.manager.controller.nifiregistryclient Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiRegistryClient", "source": "kind source: /, Kind="}
2021-11-29T17:27:09.995Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.180Z INFO controller-runtime.manager.controller.nifiusergroup Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUserGroup"}
2021-11-29T17:27:10.180Z INFO controller-runtime.manager.controller.nifiuser Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUser", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.181Z INFO controller-runtime.manager.controller.nifiregistryclient Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiRegistryClient"}
2021-11-29T17:27:10.181Z INFO controller-runtime.manager.controller.nifiparametercontext Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiParameterContext"}
2021-11-29T17:27:10.181Z INFO controller-runtime.manager.controller.nifidataflow Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiDataflow"}
2021-11-29T17:27:10.284Z INFO controller-runtime.manager.controller.nifidataflow Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiDataflow", "worker count": 1}
2021-11-29T17:27:10.294Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.381Z INFO controller-runtime.manager.controller.nificluster Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster"}
2021-11-29T17:27:10.381Z INFO controller-runtime.manager.controller.nifiusergroup Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUserGroup", "worker count": 1}
2021-11-29T17:27:10.381Z INFO controller-runtime.manager.controller.nifiuser Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUser"}
2021-11-29T17:27:10.381Z INFO controller-runtime.manager.controller.nifiregistryclient Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiRegistryClient", "worker count": 1}
2021-11-29T17:27:10.381Z INFO controller-runtime.manager.controller.nifiparametercontext Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiParameterContext", "worker count": 1}
W1129 17:27:10.385667 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 17:27:10.389258 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
2021-11-29T17:27:10.394Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.481Z INFO controller-runtime.manager.controller.nificluster Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "worker count": 1}
2021-11-29T17:27:10.495Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.582Z INFO controller-runtime.manager.controller.nifiuser Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiUser", "worker count": 1}
2021-11-29T17:27:10.596Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.697Z INFO controller-runtime.manager.controller.nificluster Starting EventSource {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "source": "kind source: /, Kind="}
2021-11-29T17:27:10.798Z INFO controller-runtime.manager.controller.nificluster Starting Controller {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster"}
2021-11-29T17:27:10.798Z INFO controller-runtime.manager.controller.nificluster Starting workers {"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "worker count": 1}
W1129 17:34:30.395634 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 17:42:32.401953 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 17:50:11.406944 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1129 17:56:23.413301 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Helm chart version : 0.7.5
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Please advise...
Thx !
From nifikop created by erdrix: Orange-OpenSource/nifikop#5
Is your feature request related to a problem? Please describe.
In the current solution, the readiness & liveness simply call the nifi-api and check if a response from nifi node is here.
Because of the need to check if the node is in the cluster to say "it is ready", this will cause issues when we decommission the node.
Describe the solution you'd like to see
As we are working at the pod level, we could imagine checking if the node is part of the cluster in normal cases, and in the case of a decommissioned node, changing the liveness and readiness script to check if we can target it?
From nifikop created by michael81877: Orange-OpenSource/nifikop#169
Is your feature request related to a problem? Please describe.
Add the ability to override/replace the default authorizers.xml template so that you can include custom authorizers, userGroupProviders, and accessPolicyProviders.
Describe the solution you'd like to see
In authorizers.go it's clear that the authorizers.xml template is there for dynamic initial admin purposes, so i think it'd be wise to continue leveraging that template. Perhaps as an extension, nifikop could allow users to override that template to provide additional authorizers, userGroupProviders, and accessPolicyProviders.
Describe alternatives you've considered
It's not currently possible to configure any other authorizers.
We must mention in the documentation that since the 1.14 of NiFi the property nifi.sensitive.props.key must be set.
And that it should be set via overrideSecretConfig.
From nifikop created by Sreenivas-Ratakonda: Orange-OpenSource/nifikop#179
After setting up the sslnifi cluster I found that the managed users are not getting created, as per the docs to login in to the Nifi cluster UI we need one admin user but that user is not getting created in the nificluster.
As per the docs by default three groups gets created managed admins, managed users, managed nodes but for me the nifi user groups are not getting created
What did you do?
I have created an sslnifi cluster.
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: sslnifi
spec:
service:
headlessEnabled: false
zkAddress: "zookeeper.zookeeper.svc.cluster.local:2181"
zkPath: "/ssllnifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: false
managedAdminUsers:
- identity : "[email protected]"
name: "nifiadmin"
managedReaderUsers:
- identity : "[email protected]"
name: "nifiuser"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
# NifiProperties configuration that will be applied to the node.
nifiProperties:
webProxyHosts:
- nifistandard2.trycatchlearn.fr:8443
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "0.5"
memory: 2Gi
requests:
cpu: "0.5"
memory: 2Gi
nodes:
- id: 1
nodeConfigGroup: "default_group"
readOnlyConfig:
nifiProperties:
overrideConfigs: |
nifi.ui.banner.text=Ciena Blueplanet Enterprise Node SSL 1
nifi.remote.input.socket.port=
nifi.remote.input.secure=true
nifi.remote.input.host=xxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com
- id: 2
nodeConfigGroup: "default_group"
readOnlyConfig:
nifiProperties:
overrideConfigs: |
nifi.ui.banner.text=Ciena Blueplanet Enterprise Node SSL 2
nifi.remote.input.socket.port=
nifi.remote.input.secure=true
nifi.remote.input.host=xxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com
- id: 3
nodeConfigGroup: "default_group"
readOnlyConfig:
nifiProperties:
overrideConfigs: |
nifi.ui.banner.text=Ciena Blueplanet Enterprise Node SSL 3
nifi.remote.input.socket.port=
nifi.remote.input.secure=true
nifi.remote.input.host=xxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com
listenersConfig:
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
sslSecrets:
tlsSecretName: "test-nifikop"
create: true
---
apiVersion: v1
kind: Service
metadata:
name: sslnifi-all
spec:
selector:
app: nifi
nifi_cr: sslnifi
ports:
- name: https
port: 8443
protocol: TCP
targetPort: 8443
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: sslnifi-1-node-lb
spec:
selector:
app: nifi
nifi_cr: sslnifi
nodeId: "1"
ports:
- name: https
port: 8443
protocol: TCP
targetPort: 8443
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: sslnifi-2-node-lb
spec:
selector:
app: nifi
nifi_cr: sslnifi
nodeId: "2"
ports:
- name: https
port: 8443
protocol: TCP
targetPort: 8443
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: sslnifi-3-node-lb
spec:
selector:
app: nifi
nifi_cr: sslnifi
nodeId: "3"
ports:
- name: https
port: 8443
protocol: TCP
targetPort: 8443
type: LoadBalancer
---
apiVersion: nifi.orange.com/v1alpha1
kind: NifiUser
metadata:
name: bpeadmin
spec:
identity: [email protected]
clusterRef:
name: sslnifi
namespace: nifi
createCert: true
includeJKS: true
secretName: bpeadmin_secrets
What did you expect to see?
we expected see managed users to be created but those users are not created in Nifi Cluster.
I have created an another user bpeadmin when I query nifikop it says user created but the user is not created in Nificluster.
few Nifi user groups needs be created.
What did you see instead? Under which circumstances?
Below we can see that there are no managed users created, which are mentioned in the Nifi Cluster config.
Here it says that bpeadmin user is created but i have added authorizer file there is no bpeadmin user created in there.
Users created in the Nifi Cluster config
$ k get nifiusers.nifi.orange.com -n nifi
NAME AGE
bpeadmin 18h
sslnifi-1-node.nifi.svc.cluster.local 18h
sslnifi-2-node.nifi.svc.cluster.local 18h
sslnifi-3-node.nifi.svc.cluster.local 18h
sslnifi-controller.nifi.mgt.cluster.local 18h
authorizers.xml file in one of the nodes.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">../data/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity admin">sslnifi-controller.nifi.mgt.cluster.local</property>
<property name="Initial User Identity 1">sslnifi-1-node.nifi.svc.cluster.local</property>
<property name="Initial User Identity 2">sslnifi-2-node.nifi.svc.cluster.local</property>
<property name="Initial User Identity 3">sslnifi-3-node.nifi.svc.cluster.local</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">../data/authorizations.xml</property>
<property name="Initial Admin Identity">sslnifi-controller.nifi.mgt.cluster.local</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">sslnifi-1-node.nifi.svc.cluster.local</property>
<property name="Node Identity 2">sslnifi-2-node.nifi.svc.cluster.local</property>
<property name="Node Identity 3">sslnifi-3-node.nifi.svc.cluster.local</property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
Detailed view at the bpeadmin user
$ k describe nifiusers.nifi.orange.com/bpeadmin -n nifi
Name: bpeadmin
Namespace: nifi
Labels: <none>
Annotations: banzaicloud.com/last-applied:
UEsDBBQACAAIAAAAAAAAAAAAAAAAAAAAAAAIAAAAb3JpZ2luYWyUk8GO0zAQht9lzk7bZHdb8AkJiQMgDrC7BwiHqT0po3Ucy3ZWWlV5d2SnSVPURXBp49HM+Pd8/xwBHT+SD9xZkG...
API Version: nifi.orange.com/v1alpha1
Kind: NifiUser
Metadata:
Creation Timestamp: 2021-12-27T12:16:18Z
Generation: 2
Managed Fields:
API Version: nifi.orange.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:banzaicloud.com/last-applied:
f:status:
.:
f:id:
f:version:
Manager: manager
Operation: Update
Time: 2021-12-27T12:16:18Z
API Version: nifi.orange.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:clusterRef:
.:
f:name:
f:namespace:
f:createCert:
f:identity:
f:includeJKS:
f:secretName:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-12-27T13:01:37Z
Resource Version: 65379941
Self Link: /apis/nifi.orange.com/v1alpha1/namespaces/nifi/nifiusers/bpeadmin
UID: 7a7b71ed-2a12-466d-9f5c-073c6b42e3a7
Spec:
Cluster Ref:
Name: sslnifi
Namespace: nifi
Create Cert: true
Identity: [email protected]
Include JKS: true
Secret Name: bpeadmin_secrets
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ReconcilingCertificate 13m (x86 over 18h) nifi-user Reconciling certificate for nifi user bpeadmin
No Nifi Groups Found
$ kubectl get -n nifi nifiusergroups.nifi.orange.com
No resources found in nifi namespace.
So to summarize there is a conflict between what we see in k get nifiusers.nifi.orange.com -n nifi and authorizers.xml one says the bpeadmin user created but the other one doesn't have the the bpeadmin user in authorizers.xml
** So over all the Users are not getting created in Nifi Cluster **
Environment
nifikop version:
Followed exact steps here:
https://orange-opensource.github.io/nifikop/docs/2_setup/1_getting_started
Kubernetes version information:
$ k version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
1.12.1
Possible Solution
Additional context
Add any other context about the problem here.
We find that nifikop will never be able to scale down the nificluster successfully if it crashes in the middle of reconcileNifiPod() and later restarts.
More concretely, inside reconcileNifiPod(), nifikop does the following:
status.nodesState[nodeId].configurationState of the nificluster cr to ConfigInSyncstatus.nodesState[nodeId].gracefulActionState.actionState of the nificluster cr to GracefulUpscaleSucceededIf nifikop crashes between 3 and 4 and later restarts, it results in an intermediate state, where the nifi pod is created (with ConfigInSync) but the corresponding actionState is not set. Note that given the pod already exists, nifikop will not run the above steps 2, 3, 4 again.
Later if the user wants to scale down the nificluster, this pod is supposed to be offloaded and deleted gracefully. Inside reconcileNifiPodDelete, nifikop checks whether the corresponding actionState of the pod is GracefulUpscaleSucceeded or GracefulUpscaleRequired. If so, it will add the pod to nodesPendingGracefulDownscale and later offload and delete the nifi node (pod). However, since the corresponding actionState is not set due to the previous crash, the graceful downscale will never happen.
What did you do?
Scale down a nificluster from 2 nodes to 1 node.
What did you expect to see?
The second nifi pod should be deleted successfully.
What did you see instead? Under which circumstances?
The second nifi pod never gets deleted.
Environment
Possible Solution
One potential solution is to switch the order of 3 (set configurationState to ConfigInSync) and 4 (set actionState to GracefulUpscaleSucceeded). If nifikop crashes before ConfigInSync is set, reconcileNifiPod() will later deletes and recreates the pod.
Additional context
We are willing to help fix the bug.
The bug is automatically found by our tool Sieve: https://github.com/sieve-project/sieve
From nifikop created by michael81877: Orange-OpenSource/nifikop#183
Is your feature request related to a problem? Please describe.
Like you can configure maximumTimerDrivenThreadCount, add ability to configure the maximumEventDrivenThreadCount. Right now it must be configured via the nifi UI.
From nifikop created by ruckc: Orange-OpenSource/nifikop#94
Is your feature request related to a problem? Please describe.
I'm always frustrated when a software package does not support the arm64 architecture.
Describe the solution you'd like to see
Release arm64 as part of your container build processes.
Additional context
arm64 is frequently found in raspberry pi kubernetes clusters and on the new Mac computers with arm64 processors.
From nifikop created by yossisht9876: Orange-OpenSource/nifikop#165
hey all,
i have a problem configure toleration to the nifi-cluster pods.
"error: error validating "nifi-cluster.yaml": error validating data: ValidationError(NifiCluster.spec): unknown field "tolerations" in com.orange.nifi.v1alpha1.NifiCluster.spec; if you choose to ignore these errors, turn validation off with --validate=false"
its works for the zookeeper setup and for the helm nifikop chart.
but i didnt find a way to set toleration or Nodeselector to the nifi-cluster.yaml
here is my conf:
kind: NifiCluster
metadata:
name: nificluster
spec:
service:
headlessEnabled: true
zkAddress: "zookeeper:2181"
zkPath: "/nifi"
clusterImage: "apache/nifi:1.13.2"
oneNifiNodePerNode: false
managedAdminUsers:
- identity: "nidddha.com"
name: "ndd.s"
- identity: "yodddusddha.dd"
name: "yddi.s"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
logbackConfig:
# logback.xml configuration that will replace the one produced based on template
replaceConfigMap:
# The key of the value,in data content, that we want use.
data: logback.xml
# Name of the configmap that we want to refer.
name: nifi-configs
# Namespace where is located the secret that we want to refer.
namespace: nifi
nifiProperties:
webProxyHosts:
- nifi.dddv.ludddsha.co
# Additionnals nifi.properties configuration that will override the one produced based
# on template and configurations.
overrideConfigs: |
xxxxxxxxxxxx
nodeConfigGroups:
default_group:
serviceAccountName: "default"
runAsUser: 1000
fsGroup: 1000
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 20Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
resourcesRequirements:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "1"
memory: 1Gi
nodes:
- id: 0
nodeConfigGroup: "default_group"
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
listenersConfig:
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
- type: "prometheus"
name: "prometheus"
containerPort: 9090
sslSecrets:
tlsSecretName: "nifi-sdds"
create: true
externalServices:
- name: "nifi-clds"
spec:
type: ClusterIP
portConfigs:
- port: 8443
internalListenerName: "https"
trying to add one of the 2:
# nodeSelector:
# nodegroup: nifi
# tolerations:
# - key: "nifi-dedicated"
# operator: "Equal"
# value: "on_demand"
# effect: "NoSchedule"
any help ? thanks @erdrix
From nifikop created by rayyno: Orange-OpenSource/nifikop#140
Is your feature request related to a problem? Please describe.
The current operation to scaledown a node through the operator is disconnecting, offloading and deleting the node.
The offloading takes too long for large data clusters when removing some nodes from the cluster, causing the cluster to get stuck until offloading is done.
Describe the solution you'd like to see
An option to choose whether to offload a node when scaling down would be great.
From nifikop created by borkod: Orange-OpenSource/nifikop#21
Hello. This is a very interesting project 👍
I am trying to follow https://orange-opensource.github.io/nifikop/blog/secured_nifi_cluster_on_gcp/ , but deploy it on Azure Kubernetes Service.
I've deployed:
I've updated the nifi cluster resource yaml file with appropriate values from above.
When I try to deploy it, I don't see any pod resources even created.
Any suggestions? What's the best way to debug why no pods are even being created? kubectl describe on the nificluster resource doesn't provide any useful information.
I was able to deploy a working cluster on AKS using simple nifi cluster sample (not secured).
Thanks for any suggestions and help!
From nifikop created by omkadmi: Orange-OpenSource/nifikop#143
What did you do?
I deployed an unsecured nifi cluster -> it works
I deployed a secure nifi cluster with a self-signed certificate (managed by nifikop) -> it works
I have deployed a secure nifi cluster with the cert manager + let'sencrypt -> does not work
I followed all the steps in the documentation https://orange-opensource.github.io/nifikop/blog/2020/06/30/secured_nifi_cluster_on_gcp_with_external_dns, but I still have this connection refused error
knowing that the certificates are issued by the cert manager
I also see the sslnifi entries in Azure private DNS created by extarnalDNS
for info, nifikop, zookeeper and the nifi cluster are in the nifi namspace, cert-manager, letsencrypt and externaldns are in the devops namespace
I have this error in the pod log (which repeats ad infinitum):
Waiting for host to be reachable
failed to reach sslnifi-0-node.sslnifi-headless.mycompany.net:8443
Found: , expecting: 10.66.161.197
Found :
failed to reach sslnifi-0-node.sslnifi-headless.mycompany.net:8443
Found: , expecting: 10.66.161.197
Found :
failed to reach sslnifi-0-node.sslnifi-headless.mycompany.net:8443
Found: , expecting: 10.66.161.197
Found :
failed to reach sslnifi-0-node.sslnifi-headless.mycompany.net:8443
Found: , expecting: 10.66.161.197
Found :
failed to reach sslnifi-0-node.sslnifi-headless.mycompany.net:8443
Found: , expecting: 10.66.161.197
Found :
failed to reach sslnifi-0-node.sslnifi-headless.mycompany.net:8443
Found: , expecting: 10.66.161.197
I have this error in the describe : Failed to connect to sslnifi-0-node.sslnifi-headless.nifi.svc.cluster.local port 8443
Readiness probe failed: * Expire in 0 ms for 6 (transfer 0x557d85ecef50) * Expire in 1 ms for 1 (transfer 0x557d85ecef50) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 0 ms for 1 (transfer 0x557d85ecef50) * Expire in 1 ms for 1 (transfer 0x557d85ecef50) * Expire in 0 ms for 1 (transfer 0x557d85ecef50) * Expire in 0 ms for 1 (transfer 0x557d85ecef50) * Expire in 0 ms for 1 (transfer 0x557d85ecef50) * Trying 10.66.161.197... * TCP_NODELAY set * Expire in 200 ms for 4 (transfer 0x557d85ecef50) * connect to 10.66.161.197 port 8443 failed: Connection refused * Failed to connect to sslnifi-0-node.sslnifi-headless.nifi.svc.cluster.local port 8443: Connection refused * Closing connection 0 curl: (7) Failed to connect to sslnifi-0-node.sslnifi-headless.nifi.svc.cluster.local port 8443: Connection refused
I don't understand why it is looking in sslnifi-0-node.sslnifi-headless.nifi.svc.cluster.local (which ends in .cluster.local) when it should (I guess) look in .mycompany.net
below the cert-manager log
31
I1018 09:49:32.621252 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-controller.nifi.mgt.mycompany.net" condition "Issuing" to 2021-10-18 09:49:32.621243958 +0000 UTC m=+1231705.057373825
30
I1018 09:49:32.622054 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-controller.nifi.mgt.mycompany.net" condition "Ready" to 2021-10-18 09:49:32.622049068 +0000 UTC m=+1231705.058179035
29
I1018 09:49:32.779307 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-0-node.sslnifi-headless.mycompany.net" condition "Issuing" to 2021-10-18 09:49:32.77929993 +0000 UTC m=+1231705.215429897
28
I1018 09:49:32.781088 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-0-node.sslnifi-headless.mycompany.net" condition "Ready" to 2021-10-18 09:49:32.781081754 +0000 UTC m=+1231705.217211621
27
E1018 09:49:32.829395 1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net"
26
I1018 09:49:32.829670 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-controller.nifi.mgt.mycompany.net" condition "Issuing" to 2021-10-18 09:49:32.829665191 +0000 UTC m=+1231705.265795058
25
E1018 09:49:32.937119 1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-0-node.sslnifi-headless.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-0-node.sslnifi-headless.mycompany.net"
24
I1018 09:49:32.937207 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-0-node.sslnifi-headless.mycompany.net" condition "Issuing" to 2021-10-18 09:49:32.937203301 +0000 UTC m=+1231705.373333268
23
E1018 09:49:32.964735 1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net"
22
I1018 09:49:32.964930 1 conditions.go:173] Setting lastTransitionTime for Certificate "sslnifi-controller.nifi.mgt.mycompany.net" condition "Issuing" to 2021-10-18 09:49:32.964925064 +0000 UTC m=+1231705.401054931
21
E1018 09:49:33.528303 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net"
20
E1018 09:49:33.913078 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-0-node.sslnifi-headless.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-0-node.sslnifi-headless.mycompany.net"
19
I1018 09:49:34.942552 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "sslnifi-controller.nifi.mgt.mycompany.net-bhx8b" condition "Ready" to 2021-10-18 09:49:34.942545097 +0000 UTC m=+1231707.378674964
18
I1018 09:49:35.375578 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw" condition "Ready" to 2021-10-18 09:49:35.375571575 +0000 UTC m=+1231707.811701442
17
I1018 09:49:36.157507 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw" condition "Ready" to 2021-10-18 09:49:36.157499429 +0000 UTC m=+1231708.593629296
16
E1018 09:49:36.712804 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw"
15
E1018 09:49:37.917592 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net-bhx8b\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net-bhx8b"
14
I1018 09:49:38.878739 1 acme.go:184] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="sslnifi-controller.nifi.mgt.mycompany.net-bhx8b-3833685911" "related_resource_namespace"="nifi" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="sslnifi-controller.nifi.mgt.mycompany.net-bhx8b" "resource_namespace"="nifi" "resource_version"="v1"
13
I1018 09:49:38.878997 1 conditions.go:222] Found status change for CertificateRequest "sslnifi-controller.nifi.mgt.mycompany.net-bhx8b" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-10-18 09:49:38.878992016 +0000 UTC m=+1231711.315121983
12
E1018 09:49:39.372677 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net"
11
I1018 09:49:39.373408 1 conditions.go:162] Found status change for Certificate "sslnifi-controller.nifi.mgt.mycompany.net" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-10-18 09:49:39.373401899 +0000 UTC m=+1231711.809531866
10
E1018 09:49:39.593733 1 controller.go:158] cert-manager/controller/CertificateIssuing "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net"
9
E1018 09:49:39.937234 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompany.net"
8
I1018 09:49:39.937992 1 conditions.go:162] Found status change for Certificate "sslnifi-controller.nifi.mgt.mycompany.net" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-10-18 09:49:39.937985336 +0000 UTC m=+1231712.374115203
7
I1018 09:49:40.134578 1 acme.go:184] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw-2332423181" "related_resource_namespace"="nifi" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw" "resource_namespace"="nifi" "resource_version"="v1"
6
I1018 09:49:40.135097 1 conditions.go:222] Found status change for CertificateRequest "sslnifi-0-node.sslnifi-headless.mycompany.net-fdrzw" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-10-18 09:49:40.135089423 +0000 UTC m=+1231712.571219390
5
E1018 09:49:40.136239 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-controller.nifi.mgt.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-controller.nifi.mgt.mycompanyv.net"
4
E1018 09:49:41.572069 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-0-node.sslnifi-headless.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-0-node.sslnifi-headless.mycompany.net"
3
I1018 09:49:41.573131 1 conditions.go:162] Found status change for Certificate "sslnifi-0-node.sslnifi-headless.mycompany.net" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-10-18 09:49:41.573123467 +0000 UTC m=+1231714.009253334
2
E1018 09:49:42.044550 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"sslnifi-0-node.sslnifi-headless.mycompany.net\": the object has been modified; please apply your changes to the latest version and try again" "key"="nifi/sslnifi-0-node.sslnifi-headless.mycompany.net"
1
I1018 09:49:42.045275 1 conditions.go:162] Found status change for Certificate "sslnifi-0-node.sslnifi-headless.mycompany.net" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-10-18 09:49:42.045269625 +0000 UTC m=+1231714.481399492
below the externaldns log
time="2021-10-13T09:35:52Z" level=info msg="Updating A record named 'sslnifi-int' to '10.66.161.134' for Azure Private DNS zone 'mycompany.net'."
time="2021-10-13T09:35:52Z" level=info msg="Updating A record named 'sslnifi-0-node.sslnifi-int' to '10.66.161.134' for Azure Private DNS zone 'mycompany.net'."
time="2021-10-13T09:35:53Z" level=info msg="Updating TXT record named 'sslnifi-int' to '\"heritage=external-dns,external-dns/owner=<server_name>,external-dns/resource=service/nifi/sslnifi-headless\"' for Azure Private DNS zone 'mycompany.net'."
time="2021-10-13T09:35:53Z" level=info msg="Updating TXT record named 'sslnifi-0-node.sslnifi-int' to '\"heritage=external-dns,external-dns/owner=<server_name>,external-dns/resource=service/nifi/sslnifi-headless\"' for Azure Private DNS zone 'mycompany.net'."
kubectl get all -n nifi
NAME READY STATUS RESTARTS AGE
pod/nifikop-int-76cbbff7c6-8fz2g 1/1 Running 0 47h
pod/sslnifi-0-nodexbwjp 0/1 Running 0 4m41s
pod/zookeeper-0 1/1 Running 0 5d1h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/clusterip ClusterIP 10.0.18.0 <none> 8443/TCP 36m
service/sslnifi-headless ClusterIP None <none> 8443/TCP,6007/TCP,10000/TCP 36m
service/zookeeper ClusterIP 10.0.18.121 <none> 2181/TCP,2888/TCP,3888/TCP 5d1h
service/zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 5d1h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nifikop-int 1/1 1 1 47h
NAME DESIRED CURRENT READY AGE
replicaset.apps/nifikop-int-76cbbff7c6 1 1 1 47h
NAME READY AGE
statefulset.apps/zookeeper 1/1 5d1h
below my conf:
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: sslnifi
namespace: nifi
spec:
service:
headlessEnabled: true
annotations:
external-dns.alpha.kubernetes.io/ttl: "60"
zkAddress: "zookeeper:2181"
zkPath: "/sslnifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: false
managedAdminUsers:
- identity : "[email protected]"
name: "myname"
managedReaderUsers:
- identity : "[email protected]"
name: "toto"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 20
readOnlyConfig:
nifiProperties:
webProxyHosts:
- sslnifi-int.mycompany.net
# Additionnals nifi.properties configuration that will override the one produced based
# on template and configurations.
overrideConfigs: |
nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration
nifi.security.user.oidc.client.id=xxxxxxxxxxxxxxxxxxx
nifi.security.user.oidc.client.secret=xxxxxxxxxxxxxxxxxx
nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "default"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "default"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "default"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "default"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "default"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "default"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "2"
memory: 3Gi
requests:
cpu: "1"
memory: 1Gi
nodes:
- id: 0
nodeConfigGroup: "default_group"
listenersConfig:
useExternalDNS: true
clusterDomain: "mycompany.net"
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
sslSecrets:
tlsSecretName: "sslnifi-int.mycompany.net-tls"
create: true
clusterScoped: true
issuerRef:
kind: ClusterIssuer
name: letsencrypt-staging
externalServices:
- name: "clusterip"
spec:
type: ClusterIP
portConfigs:
- port: 8443
internalListenerName: "https"
serviceAnnotations:
toto: tata
I deployed the nifikop with
helm repo add orange-incubator https://orange-kubernetes-charts-incubator.storage.googleapis.com/
helm install nifikop \
orange-incubator/nifikop \
--namespace=nifi \
--version 0.7.0 \
--set image.tag=v0.7.0-release \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set namespaces={"nifi"}
I thank you in advance for your help, I've been working on it for a few days, and I don't see any solutions
What did you expect to see?
the pod of the nifi node must have the running stattus 1/1
What did you see instead? Under which circumstances?
the nifi node pod is running 0/1
Environment
0.7.0
the same problem with 0.6.3
V1.19.11
Kubernetes cluster kind:
NiFi version:
1.12.1
The documentation is inaccurate, we should not mention debug.enabled anymore.
We need to replace it with logLevel.
https://github.com/konpyutaika/nifikop/blob/master/site/docs/2_setup/3_install/1_customizable_install_with_helm.md?plain=1#L45
https://github.com/konpyutaika/nifikop/blob/master/site/docs/2_setup/3_install/1_customizable_install_with_helm.md?plain=1#L90
From nifikop created by erdrix: Orange-OpenSource/nifikop#4
Is your feature request related to a problem? Please describe.
Regarding multi cluster k8s situation, it would be great if the operator could support multi-kubernetes deployment. It should allow :
Describe the solution you'd like to see
We have at least two ways of doing so :
Is your feature request related to a problem? Please describe.
Prometheus metrics are already exposed via the /metrics path for each nifi pod via the NifiCluster.Spec.ListenersConfig: https://konpyutaika.github.io/nifikop/docs/5_references/1_nifi_cluster/6_listeners_config
add configuration to the NifiCluster spec to optionally create a ServiceMonitor resource so that metrics can be auto-scraped by Prometheus.
Describe the solution you'd like to see
A new configuration is added to the NifiCluster spec to optionally create a service monitor:
spec:
serviceMonitor:
enabled: trueHere's the rationale and behavior of Prometheus Operator ServiceMonitor resources:
Describe alternatives you've considered
A ServiceMonitor is easily created outside of nifikop, but a simple configuration and code change can have nifikop deploy the resource for you.
From nifikop created by iordaniordanov: Orange-OpenSource/nifikop#139
About general context and help around nifikop
What did you do?
Increased number of nodes in the nificluster CR from 3 to 6
What did you expect to see?
3 new nodes to be simultaneously created and joined in the cluster
What did you see instead? Under which circumstances?
3 new nodes were simultaneously created, they join the cluster, but after that they are one by one re-created and only after that the cluster is fully functional, which leads to a linear increase in the amount of time which is needed to scale the cluster up. If adding one node takes 5 min adding 2 nodes takes ~10 min and so on. Is this the expected behavior or it is an issue with our configuration/environment ?
Environment
nifikop version:
v0.6.0
Kubernetes version information:
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-eks-087e67", GitCommit:"087e67e479962798594218dc6d99923f410c145e", GitTreeState:"clean", BuildDate:"2021-07-31T01:39:55Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Kubernetes cluster kind:
EKS
NiFi version:
1.12.1
Additional context
Nifi cluster config
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: <name>
namespace: <namespace>
spec:
clusterImage: <image> # Nifi 1.12.1 image
externalServices:
- name: clusterip
spec:
portConfigs:
- internalListenerName: http
port: 8080
type: ClusterIP
initContainerImage: <busybox image>
listenersConfig:
internalListeners:
- containerPort: 8080
name: http
type: http
- containerPort: 6007
name: cluster
type: cluster
- containerPort: 10000
name: s2s
type: s2s
- containerPort: 9090
name: prometheus
type: prometheus
nifiClusterTaskSpec:
retryDurationMinutes: 10
nodeConfigGroups:
default_group:
isNode: true
resourcesRequirements:
limits:
cpu: "2"
memory: 6Gi
requests:
cpu: "2"
memory: 6Gi
serviceAccountName: default
storageConfigs:
- mountPath: /opt/nifi/data
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: general
- mountPath: /opt/nifi/content_repository
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: general
- mountPath: /opt/nifi/flowfile_repository
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: general
- mountPath: /opt/nifi/provenance_repository
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: general
- mountPath: /opt/nifi/nifi-current/work
name: work
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: general
nodes:
- id: 0
nodeConfigGroup: default_group
- id: 1
nodeConfigGroup: default_group
- id: 2
nodeConfigGroup: default_group
oneNifiNodePerNode: true
propagateLabels: true
readOnlyConfig:
bootstrapProperties:
nifiJvmMemory: 2g
overrideConfigs: |
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
conf.dir=./conf
nifiProperties:
overrideConfigs: |
nifi.nar.library.autoload.directory=./extensions
nifi.web.http.network.interface.default=eth0
nifi.web.http.network.interface.lo=lo
nifi.web.proxy.context.path=<proxy_path>
nifi.database.directory=/opt/nifi/data/database_repository
nifi.flow.configuration.archive.dir=/opt/nifi/data/archive
nifi.flow.configuration.file=/opt/nifi/data/flow.xml.gz
nifi.templates.directory=/opt/nifi/data/templates
nifi.provenance.repository.max.storage.size=2GB
nifi.provenance.repository.indexed.attributes=te$containerId,te$id
webProxyHosts:
- <proxy_host>
zookeeperProperties: {}
service:
headlessEnabled: true
zkAddress: <zk_addr>
zkPath: <zk_path>
From nifikop created by Premfer: Orange-OpenSource/nifikop#122
Hi,
I deployed nifikop using nifi , all are deployed correctly but UI says flow controller is still initializing. The log of nifiKop says
Wed, Aug 11 2021 7:04:38 pm | 2021-08-11T13:34:38.780Z INFO nifi_client 404 response from nifi node: 404 Not Found
Wed, Aug 11 2021 7:04:38 pm | 2021-08-11T13:34:38.780Z INFO controllers.NifiCluster Nodes unreachable, may still be starting up
Can you let know what is the thing need to be checked ?
From nifikop created by riccardo-salamanna: Orange-OpenSource/nifikop#141
Testing of nifikop with version 1.14.0 have been unsuccessful.
Describe the solution you'd like to see
It would be nice since to have that support, since the release fixes some important bugs for us
Many thanks
From nifikop created by erdrix: Orange-OpenSource/nifikop#6
Is your feature request related to a problem? Please describe.
The current readiness & liveness for a NiFi node are based on the possibility to query the port, but we don't challenge if the node is connected and part of the cluster. This limitation is introduced by the fact that in a scale down situation, the pod will be detected as not Ready, and lead to a "freeze" situation. Nonetheless, this case is not the best one ...
Describe the solution you'd like to see
It would be interesting, as we work on pod and not on Statefulset, to apply a different liveness & Readiness in a case of scale down for the targeted node (and only for it).
PodDisruptionBudget v1beta1 is deprecated in Kubernetes 1.21 and will be removed in 1.25. Update nifikop to use v1 instead of v1beta1.
From nifikop created by erdrix: Orange-OpenSource/nifikop#117
What did you do?
Tried to update a NiFiParameterContext configuration, with ControllerServices referencing some of it parameters.
What did you expect to see?
A clean and successful parameter context update
What did you see instead? Under which circumstances?
The update was blocked due to some active controller services.
Environment
Possible Solution
Test if the current update is blocked by some controller constraints, if so, disable the controller service until the parameter context successfully update.
From nifikop created by erdrix: Orange-OpenSource/nifikop#10
Is your feature request related to a problem? Please describe.
The aim of this feature is to allow users of the helm chart providing custom annotations for operator deployment. Via annotations, users can configure metadata processed by other parties like logs collectors, kube-janitor TTL etc...
From nifikop created by erdrix: Orange-OpenSource/nifikop#2
Is your feature request related to a problem? Please describe.
If we want to be able to debug operator or cluster's nodes, we need to use kubectl alpha debug command with ephemeral containers instead of kubectl exec. To do so, we need to enable shareProcessNamespace into pods.
Describe the solution you'd like to see
Add this field into the CRD and in chart template.
Additional context
Check with security team if there are any restrictions about letting it enabled !
From nifikop created by michael81877: Orange-OpenSource/nifikop#149
Is your feature request related to a problem? Please describe.
No.
Describe the solution you'd like to see
I'd like to attach arbitrary labels to pods that nifikop creates. Specifically, i'd like to label pods with meaningful things as a means to select pods for log aggregation via https://rancher.com/docs/rancher/v2.6/en/logging/
For example:
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: simplenifi
spec:
...
pod:
labels:
specific-pod-label: "abc"
nodeConfigGroup:
default_group:
nodeLabels:
my-special-label: "true"
...
Describe alternatives you've considered
Annotations can't be used as pod selectors and the only labels applied to the pods nifikop creates match the name of the NiFiCluster CRD under the label nifi_cr. Otherwise, there's not a label that distinguishes multiple nifi clusters deployed by nifikop.
From nifikop created by nnachefski: Orange-OpenSource/nifikop#192
Attempting to deploy simplenificluster.yaml on Openshift 4.8 and am hitting this error in the operator pod:
{"reconciler group": "nifi.orange.com", "reconciler kind": "NifiCluster", "name": "simplenifi", "namespace": "nifi", "error": "failed to reconcile resource: creating resource failed: services \"simplenifi-headless\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"
# kubectl get role
NAME CREATED AT
nifikop 2022-02-10T09:59:01Z
# kubectl get rolebindings
NAME ROLE AGE
nifikop Role/nifikop 30m
# kubectl get sa nifikop
NAME SECRETS AGE
nifikop 2 30m
# kubectl get deploy nifikop -o 'jsonpath={.spec.template.spec.serviceAccountName}'
nifikop
Any ideas on why this might be happening on Openshift kube?
From nifikop created by michael81877: Orange-OpenSource/nifikop#151
The NiFiCluster.spec.nodeConfigGroup.nodeAnnotations documented here actually map to NiFiCluster.spec.nodeConfigGroup.nifiAnnotations in the custom resource definition.
The name would be more appropriate as nodeAnnotations given its location in the CRD.
source of the problem is here: https://github.com/Orange-OpenSource/nifikop/blob/master/api/v1alpha1/nificluster_types.go#L284
Environment
Kubernetes cluster kind:
NiFi version:
Possible Solution
I've submitted a PR with a related change and renamed NiFiCluster.spec.nodeConfigGroup.nifiAnnotations to NiFiCluster.spec.nodeConfigGroup.nodeAnnotations to match the site documentation.
Additional context
I've submitted a fix for this in #150
From nifikop created by michael81877: Orange-OpenSource/nifikop#147
Is your feature request related to a problem? Please describe.
No.
Describe the solution you'd like to see
Add ability to attach additional volumes to the NiFi cluster image to mount arbitrary configs into the pod :
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: simplenifi
spec:
...
volumes:
- name: secret-params-volume
secret:
secretName: secret-params
volumeMounts:
- mountPath: /params
name: secret-params-volume
...
Describe alternatives you've considered
The only possible alternative right now is to re-use the existing node-tmp volume via an init container, but you can't pull arbitrary secrets and/or config maps into it. This feature enables that.
From nifikop created by sanamart: Orange-OpenSource/nifikop#185
I would like to access the slack channel but i can not do it without an orange account. Can anyone grant me access with a different email address?
Thanks.
From nifikop created by riccardo-salamanna: Orange-OpenSource/nifikop#159
SSL access for api user support
Our cluster is currently configured to have authentication via OIDC and it works fine but we would like to have a user for api access, authenticating via certificates. We followed the steps listed at https://orange-opensource.github.io/nifikop/docs/3_tasks/2_security/1_ssl#create-ssl-credentials, extrated the cert but we are still not able to authenticate with the provisioned user and the certificates that have been generated. Is there any step missing? thanks
Environment
Environment
nifikop version: 0.7.1
Kubernetes version information: 1.19
Kubernetes cluster kind: EKS
NiFi version: 1.13.2
From nifikop created by tmarkunin: Orange-OpenSource/nifikop#69
What did you do?
I've installed simple nifi cluster following https://orange-opensource.github.io/nifikop/docs/2_setup/1_getting_started
What did you expect to see?
Running nifi cluster with 2 nodes accessible through web UI
NAME READY STATUS RESTARTS AGE
pod/nifikop-586867994d-lkmgc 1/1 Running 0 6h56m
pod/nifikop-586867994d-pvnmn 0/1 Terminating 0 25h
pod/simplenifi-1-nodew5925 1/1 Running 0 6h52m
pod/simplenifi-2-nodegt8rh 1/1 Running 0 22h
pod/zookeeper-0 1/1 Running 1 6h52m
pod/zookeeper-1 1/1 Running 1 6h52m
pod/zookeeper-2 1/1 Running 1 6h52m
What did you see instead? Under which circumstances?
UI is not accessible through svc service/simplenifi-all-node. Moreover I failed to curl http:localhost:8080 from inside a container
$ curl http://localhost:8080/nifi
curl: (7) Failed to connect to localhost port 8080: Connection refused
Environment
nifikop version: 0.5.1
Kubernetes version information:
1.18
From nifikop created by srteam2020: Orange-OpenSource/nifikop#130
What did you do?
We find that the NiFi operator sometimes fails to update the config specified by the user if it crashes and restarts at some particular point.
More concretely, the user can update the NiFi config by patching/updating the NifiCluster CR (e.g. set nifiProperties to nifi.ui.banner.text=xxx). Ideally, the NiFi operator will update the config delta to configmap, and restart the related NiFi pod later. Detailed steps are listed here:
configmap data to make it consistent with the CR specConfigOutofSync to CR statusConfigOutofSync is set. If true, it deletes the pod (nifi node), and restarts it in next round of reconcileConfigInSync.However, we find that if the operator crashes between step 2 and step 3 and restarts, the NiFi pod will never get restarted and the new config will never be populated to NiFi correctly. The reason is that after step 2, the configmap is already consistent with the CR spec, but the CR status is not set to ConfigOutofSync yet. After restart, the operator finds there is no change in step 1, and will never try to set ConfigOutofSync again.
What did you expect to see?
The NiFi pod starts to use the new config.
What did you see instead? Under which circumstances?
The new config is not reloaded successfully.
Environment
Possible Solution
A simple fix to this would be switching step 2 and step 3. In that case, the operator still knows the config is out of sync even if it crashes at the middle, and the config will be reloaded after the operator restarts.
Additional context
We are willing to send a PR to help fix this bug.
From nifikop created by jrmanes: Orange-OpenSource/nifikop#126
How can I update the config files (nifi.properties, state-management.xml, zookeeper.properties, bootstrap-notification-servces.xml, bootstrap.conf, logback.xml, login-identity-providers.xml)? I mean, with the latest update PR-87 it creates a secret with all the config files in base64, if I need to modify any value of that, how can I do it?
What did you do?
I need to being able to update the config files.
What did you expect to see?
update the configmaps or secrets (I don't know exactly which one) and they will be update inside the pods next deploy.
What did you see instead? Under which circumstances?
There is no change, it keeps the version inside the secrets.
Environment
nifikop version: v0.6.3-release
Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-13T15:45:10Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"f3add640dbcd4f3c33a7749f38baaac0b3fe810d", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.16) exceeds the supported minor version skew of +/-1
From nifikop created by ayush7007: Orange-OpenSource/nifikop#180
I am integrating NIFI with ldap however can only define below option ( available under "LdapConfiguration" )-
This would need other properties as well to integrate successfully such as “manager dn”, “manager password”, “authentication strategy”, etc.
I don’t see any option available to provide these Ldap properties. Also overriding “login-identity-providers.xml” is not available.
Please let me know the way to provide these values, or whether it's available ?
From nifikop created by jrmanes: Orange-OpenSource/nifikop#123
Is your feature request related to a problem? Please describe.
It will be great in order to change the command from the manifest due to I need to execute some commands inside the container.
Describe the solution you'd like to see
Add an option to check if there is command from the manifest (spec).
Describe alternatives you've considered
Doing it with an initcontainer, but it will have to mount the nifi home path.
From nifikop created by michael81877: Orange-OpenSource/nifikop#153
As of cert-manager v1.6.0, it no longer publishes API versions v1alpha2, v1alpha3, and v1beta1. nifikop should be updated to support API v1, which was introduced in cert-manager v0.16:
https://github.com/jetstack/cert-manager/releases/tag/v1.6.0
This means you cannot run cert-manager v1.6.0 or later with nifikop.
Environment
Kubernetes cluster kind:
NiFi version: 1.14.0
Possible Solution
Update nifikop to support cert-manager v1 CRDs
From nifikop created by vipmaha1: Orange-OpenSource/nifikop#48
Unable to register with NifiKop Slack workspace?
What did you do?
In community support, visit below URL:
https://nifikop.slack.com/
Need cred to register to work space.
What did you expect to see?
Expecting Nifikop workspace to get added to Skack, but credentials didn't work.
What did you see instead? Under which circumstances?
Its reporting my email id is not registered.
Environment
nifikop version:
insert release or Git SHA here
Kubernetes version information:
insert output of kubectl version here
Kubernetes cluster kind:
NiFi version:
Additional context
Add any other context about the question here.
From nifikop created by yossisht9876: Orange-OpenSource/nifikop#177
Hi Guys
is there add way to TAG the nifi cluster pods ?
or add ENV variables to the nifi pods like env = dev\prod ?
here is my conf:
metadata:
name: nificluster
spec:
service:
headlessEnabled: true
zkAddress: "zookeeper:2181"
zkPath: "/nifi"
clusterImage: "apache/nifi:1.13.2"
oneNifiNodePerNode: false
managedAdminUsers:
- identity: "nidddha.com"
name: "ndd.s"
- identity: "yodddusddha.dd"
name: "yddi.s"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
logbackConfig:
# logback.xml configuration that will replace the one produced based on template
replaceConfigMap:
# The key of the value,in data content, that we want use.
data: logback.xml
# Name of the configmap that we want to refer.
name: nifi-configs
# Namespace where is located the secret that we want to refer.
namespace: nifi
nifiProperties:
webProxyHosts:
- nifi.dddv.ludddsha.co
# Additionnals nifi.properties configuration that will override the one produced based
# on template and configurations.
overrideConfigs: |
xxxxxxxxxxxx
nodeConfigGroups:
default_group:
serviceAccountName: "default"
runAsUser: 1000
fsGroup: 1000
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 20Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 50Gi
resourcesRequirements:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "1"
memory: 1Gi
nodes:
- id: 0
nodeConfigGroup: "default_group"
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
listenersConfig:
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
- type: "prometheus"
name: "prometheus"
containerPort: 9090
sslSecrets:
tlsSecretName: "nifi-sdds"
create: true
externalServices:
- name: "nifi-clds"
spec:
type: ClusterIP
portConfigs:
- port: 8443
internalListenerName: "https"
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.