eshepelyuk / cmak-operator Goto Github PK

Currently, in order to enable authentication on cmak with this chart, you can do the following:

ui:
  # enable basic auth on the UI
  extraArgs:
    - "-DbasicAuthentication.enabled=true"
    - "-DbasicAuthentication.username=<username>"
    - "-DbasicAuthentication.password=<password>"

This results in the password being stored on the manifest within the cluster, which enables anyone with read access to get the manifest AND the password for cmak. It would be ideal if something similar to the below could be allowed such that the chart was extended in capabilities like so:

---
# Values file
authentication:
  enabled: false
  username: ""
  password: ""

I'll open a PR to show this as I really want to avoid exposing a secret via a manifest

cmak-operator version: 0.5.0

Hi,

I want to make Kafka manager vailable at /kmanager/.
There for I have added - "-Dplay.http.context=/kmanager" to the values.yaml.

camp-setup is repeatingly failing with:
2021-04-28 09:40:33,005 Connecting to cmak.cmak-ns(10.105.231.20):2181, use_ssl: False 2021-04-28 09:40:43,016 Connection dropped: socket connection error: None 2021-04-28 09:40:43,101 Connecting to cmak.cmak-ns(10.105.231.20):2181, use_ssl: False 2021-04-28 09:40:53,112 Connection dropped: socket connection error: None 2021-04-28 09:40:53,213 Failed connecting to Zookeeper within the connection retry policy. 2021-04-28 09:40:53,213 Zookeeper session closed, state: CLOSED Traceback (most recent call last): File "/app/bin/cmak2zk.py", line 63, in <module> cmak2zk() File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/app/bin/cmak2zk.py", line 39, in cmak2zk zk.start() File "/usr/local/lib/python3.8/site-packages/kazoo/client.py", line 635, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.threading.KazooTimeoutError: Connection time-out

logs from zk and ui container in camp pod looks fine, but for ui pod, the health check is failing with:
Liveness probe failed: HTTP probe failed with statuscode: 404

The CMAK application has the following line:

val jmxPortResult = fieldExtended[Int]("jmx_port")(json)

It would be nice to be able to configure the JMX port in the config file. Not every Kafka is exposing JMX on TCP 12345.

Hi, Nice project :)

I want to put the CMAK in read only mode to change:

application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]

for
application.features=["KMPreferredReplicaElectionFeature”,“KMReassignPartitionsFeature"] and vice-versa.

What do you think about (overwrite application.conf option )

there is another wat to put CMAK in read only mode ?

Thanks a lot !!!

Annotation to specify ingress class, kubernetes.io/ingress.class, has been deprecated for quite some time and it is recommended to use spec.ingressClassName instead.

This would just require to add one field in https://github.com/eshepelyuk/cmak-operator/blob/master/templates/ingress.yaml

Hi guys!

We would like to configure basic authentication without expose the user and password in the values. We have a integration with Vault secrets management that injects secret automatically when the pods are starting.

I have been thinking in two ways to avoid expose user and password in values.

1. Give a new configuration on values to inform the extra Envs.

ui:
  extraEnvs:
    KAFKA_MANAGER_USERNAME: vault:/secrets/data/cmak#username
    KAFKA_MANAGER_PASSWORD: vault:/secrets/data/cmak#password

2. Give a new configuration on values to generate a new k8s secret and bind it with deployment.

ui:
  secretName: cmak-secret

apiVersion: v1
kind: Secret
metadata:
  name: cmak-secret
  labels:
type: Opaque
data:
  username: vault:/secrets/data/cmak#username
  password: vault:/secrets/data/cmak#password

What do you think?

Thanks in advanced.

Hi I have an existing Kafka cluster with ZooKeeper cluster in EKS in AWS. (I built this cluster by using Strimzi operator and Kafka Kind). I failed to add this existing cluster to CMAK, after I successfully installed CMAK by cmak-operator chart. Is there anything wrong with my procedure?

kubectl -n kafka get pods
NAME                                                            READY   STATUS    RESTARTS        AGE
cn-northwest-1-ops-kafka-crd-entity-operator-546c96d747-nggbz   3/3     Running   0               38s
cn-northwest-1-ops-kafka-crd-kafka-0                            1/1     Running   0               2m41s
cn-northwest-1-ops-kafka-crd-kafka-1                            1/1     Running   0               2m41s
cn-northwest-1-ops-kafka-crd-kafka-2                            1/1     Running   0               2m41s
cn-northwest-1-ops-kafka-crd-zookeeper-0                        1/1     Running   3 (3m44s ago)   5m45s
cn-northwest-1-ops-kafka-crd-zookeeper-1                        1/1     Running   4 (3m4s ago)    5m45s
cn-northwest-1-ops-kafka-crd-zookeeper-2                        1/1     Running   0               5m45s
strimzi-cluster-operator-5c8c984b7b-tcfnt                       1/1     Running   0               4h22m

From the above screenshot, I added zookeeper hosts info into the second box.
cn-northwest-1-ops-kafka-crd-zookeeper-0:2181, cn-northwest-1-ops-kafka-crd-zookeeper-1:2181, cn-northwest-1-ops-kafka-crd-zookeeper-2:2181/kafka

Then, got timeout error:

With docker pull limits these days, it would be nice if the image could be pulled from an alternate repository provided in values.

Hello!
Due to zookeeper has no storage, and cluster config is applied only once on creation via job, cluster list is not persistent and will disappear after every CMAK pod recreation.
There is 2 possible ways to solve this:

1. Add persistent storage to Zookeeper.
2. Run cluster population script in an initContainer.
   Thanks.

We are using managed K8s installation in Google Cloud Platform (GKE) and chart installation fails with

Error: INSTALLATION FAILED: chart requires kubeVersion: >=1.21 which is incompatible with Kubernetes v1.21.5-gke.1302

Looking at the code, it's possible to parametrise Ingress object, based on K8s version using Capabilities built-in object, so kubeVersion could be dropped.

Roughly, this would look like the following:

templates/_utils.tpl:

...

{{- define "kubeVersion" -}}
  {{- default .Capabilities.KubeVersion.Version (regexFind "v[0-9]+\\.[0-9]+\\.[0-9]+" .Capabilities.KubeVersion.Version) -}}
{{- end -}}

{{- define "ingress.apiVersion" -}}
  {{- if and (.Capabilities.APIVersions.Has "networking.k8s.io/v1") (semverCompare ">= 1.19.x" (include "kubeVersion" .)) -}}
    {{- print "networking.k8s.io/v1" -}}
  {{- else if .Capabilities.APIVersions.Has "networking.k8s.io/v1beta1" -}}
    {{- print "networking.k8s.io/v1beta1" -}}
  {{- else -}}
    {{- print "extensions/v1beta1" -}}
  {{- end -}}
{{- end -}}

templates/ingress.yaml

apiVersion: {{ template "ingress.apiVersion" . }}
...

That would make chart more versatile. If this approach is fine with you, I can open a PR.

Дякую / Thanks.

Is there an option to change the JMX port to a custom port? I use AWS MSK and the default JMX port is 1101. where can I change that?

Hi there,

I am trying to troubleshoot an issue where the Consumer data is missing when we add one particular cluster to the pool.

In the UI container logs I see

[ERROR] k.m.a.c.KafkaAdminClientActor - AdminClient not initialized yet, cannot process request : KAGetGroupSummary(ArrayBuffer(),[])

Once we remove this particular cluster from the pool of clusters, all data related to consumer for all clusters are visible.

Any help with debugging this issue is appreciated.

Version used: 2.2.0

i see you are deleting images from dockerhub. last week i was still able to install 1.3.0 but now images are deleted from there. can we set a longer retention period there? keeping last 2 majors can result in chart breaking every couple of weeks. this will not make this really available for production use....

In cases where global values are provided or if this chart is included as a subchart (even if no globals are actually specified), the values schema is overly restrictive and fails validation.

Hi CMAK team!

I'm implementing this operator right now and I'm having issues on activating LDAP. Here is the chart I'm using to configure that.

# various settings for CMAK UI container and CMAK application
ui:
  port: 9000
  # additional command line arguments
  extraArgs:
    - "-DbasicAuthentication.enabled=true"
    - "-DbasicAuthentication.ldap.enabled=true"
    - "-DbasicAuthentication.ldap.server=corpdsuat.company.com"
    - "-DbasicAuthentication.ldap.port=389"
    - "-DbasicAuthentication.ldap.username=uid=$capturedLogin$,ou=people,o=company.com,o=email"
    - "-DbasicAuthentication.ldap.password='$capturedPassword'"
    - "-DbasicAuthentication.ldap.search-base-dn=ou=people,o=company.com,o=email"
    - "-DbasicAuthentication.ldap.connection-pool-size=10"
    - "-DbasicAuthentication.ldap.ssl=false"
    - "-DbasicAuthentication.ldap.ssl-trust-all=true"
    - "-DbasicAuthentication.ldap.starttls=false"

With this code, when I get to the URL, it prompts me for user and password. Once I enter them I get the "Oops, an error occurred" and on the pod logs I get:

! @8243ilg5n - Internal server error, for (GET) [/] ->
 
play.api.UnexpectedException: Unexpected exception[LDAPBindException: invalid credentials]
	at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
	at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:178)
	at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:382)
	at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:380)
	at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
Caused by: com.unboundid.ldap.sdk.LDAPBindException: invalid credentials
	at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2273)
	at com.unboundid.ldap.sdk.LDAPConnection.<init>(LDAPConnection.java:670)
	at com.unboundid.ldap.sdk.LDAPConnection.<init>(LDAPConnection.java:563)
	at controllers.LDAPAuthenticator.ldapConnectionPool$lzycompute(BasicAuthenticationFilter.scala:156)
	at controllers.LDAPAuthenticator.ldapConnectionPool(BasicAuthenticationFilter.scala:144)
	at controllers.LDAPAuthenticator.isAuthorized(BasicAuthenticationFilter.scala:195)
	at controllers.LDAPAuthenticator.checkAuthentication(BasicAuthenticationFilter.scala:169)
	at controllers.BasicAuthenticationFilter.apply(BasicAuthenticationFilter.scala:30)
	at play.api.mvc.Filter$$anon$2.apply(Filters.scala:63)
	at play.api.mvc.Filter$$anon$2.apply(Filters.scala:50)
2023-06-28 13:47:30,569 - [ERROR] application - 

The credentials are not invalid. I have just changed and tested it on another application that uses the same LDAP server. That one is working fine.

Is there anything I have to change on the username and password configuration for the chart?

Thank you very much!

Helm style values for

• ZK
• CMAK
• cmak2zk

First off all great work (=

I'm currently trying to debug the connection from cmake-operator to zookeeper.

Version: 1.4.2

First my values config looked like this:

  21     cmak:
  22       clusters:
  23         - name: "stage"
  24           kafkaVersion: "2.7.0"
  25           curatorConfig:
  26             zkConnect: "stage-zookeeper-client:2181"

For debugging I tried to add some extraArgs to the ui like:

  21     cmak:
  22       clusters:
  23         - name: "stage"
  24           kafkaVersion: "2.7.0"
  25           curatorConfig:
  26             zkConnect: "stage-zookeeper-client:2181"
  27       ui:
  28         extraArgs:
  29           - "-Djavax.net.debug=ssl,handshake"

As I try to update the helm chart I get following message:

Helm upgrade failed: values don't meet the specifications of the schema(s) in the following chart(s):\ncmak-operator:\n- cmak: Additional property ui is not allowed

I'm trying to use this helm chart, but it seem that there is a problem:

# helm repo add cmak-operator https://eshepelyuk.github.io/cmak-operator/
Error: looks like "https://eshepelyuk.github.io/cmak-operator/" is not a valid chart repository or cannot be reached: failed to fetch https://eshepelyuk.github.io/cmak-operator/index.yaml : 404 Not Found

It's quite perplexing that I seem to be the only one experiencing this issue. I've made attempts to resolve it on multiple computers where I have Helm installed, yet the error persists consistently across all of them

Thanks in advance.

how can I pass array like basicAuthentication.excluded=["/cmak/api/health"] in ui.extraArgs ? seems everyting will convert to string

The latest version supports new versions of kafka
yahoo/CMAK#871

later deployments of the chart currently fail due to job not being able to update properly...

found this solution here

i will implement with proper values as soon as i can...

ii is my understanding that due to the limitations of cmak, which specifies that 1 consumer properties object will apply to all clusters effectively requiring allowing the same certificate across all managed clusters, is not the best solution. i do need this in my implementation.
so the question is, can we add this support here and allow passing consumer properties to the chart using it's values. if this is accepted i will add the support on my open fork and open the respective PR. WDYT?

I want to change the context path of the UI to work behind ingress.
is it possible to expose the "play.http.context" in values.yaml?