eshepelyuk / cmak-operator Goto Github PK
View Code? Open in Web Editor NEWCMAK (prev. Kafka Manager) for Kubernetes
License: MIT License
CMAK (prev. Kafka Manager) for Kubernetes
License: MIT License
Currently, in order to enable authentication on cmak
with this chart, you can do the following:
ui:
# enable basic auth on the UI
extraArgs:
- "-DbasicAuthentication.enabled=true"
- "-DbasicAuthentication.username=<username>"
- "-DbasicAuthentication.password=<password>"
This results in the password being stored on the manifest within the cluster, which enables anyone with read
access to get the manifest AND the password for cmak. It would be ideal if something similar to the below could be allowed such that the chart was extended in capabilities like so:
---
# Values file
authentication:
enabled: false
username: ""
password: ""
I'll open a PR to show this as I really want to avoid exposing a secret via a manifest
cmak-operator version: 0.5.0
Hi,
I want to make Kafka manager vailable at /kmanager/.
There for I have added - "-Dplay.http.context=/kmanager" to the values.yaml.
camp-setup is repeatingly failing with:
2021-04-28 09:40:33,005 Connecting to cmak.cmak-ns(10.105.231.20):2181, use_ssl: False 2021-04-28 09:40:43,016 Connection dropped: socket connection error: None 2021-04-28 09:40:43,101 Connecting to cmak.cmak-ns(10.105.231.20):2181, use_ssl: False 2021-04-28 09:40:53,112 Connection dropped: socket connection error: None 2021-04-28 09:40:53,213 Failed connecting to Zookeeper within the connection retry policy. 2021-04-28 09:40:53,213 Zookeeper session closed, state: CLOSED Traceback (most recent call last): File "/app/bin/cmak2zk.py", line 63, in <module> cmak2zk() File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/app/bin/cmak2zk.py", line 39, in cmak2zk zk.start() File "/usr/local/lib/python3.8/site-packages/kazoo/client.py", line 635, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.threading.KazooTimeoutError: Connection time-out
logs from zk and ui container in camp pod looks fine, but for ui pod, the health check is failing with:
Liveness probe failed: HTTP probe failed with statuscode: 404
The CMAK application has the following line:
val jmxPortResult = fieldExtended[Int]("jmx_port")(json)
It would be nice to be able to configure the JMX port in the config file. Not every Kafka is exposing JMX on TCP 12345.
Hi, Nice project :)
I want to put the CMAK in read only mode to change:
application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]
for
application.features=["KMPreferredReplicaElectionFeature”,“KMReassignPartitionsFeature"] and vice-versa.
What do you think about (overwrite application.conf option )
there is another wat to put CMAK in read only mode ?
Thanks a lot !!!
Annotation to specify ingress class, kubernetes.io/ingress.class
, has been deprecated for quite some time and it is recommended to use spec.ingressClassName
instead.
This would just require to add one field in https://github.com/eshepelyuk/cmak-operator/blob/master/templates/ingress.yaml
Hi guys!
We would like to configure basic authentication without expose the user and password in the values. We have a integration with Vault secrets management that injects secret automatically when the pods are starting.
I have been thinking in two ways to avoid expose user and password in values.
ui:
extraEnvs:
KAFKA_MANAGER_USERNAME: vault:/secrets/data/cmak#username
KAFKA_MANAGER_PASSWORD: vault:/secrets/data/cmak#password
ui:
secretName: cmak-secret
apiVersion: v1
kind: Secret
metadata:
name: cmak-secret
labels:
type: Opaque
data:
username: vault:/secrets/data/cmak#username
password: vault:/secrets/data/cmak#password
What do you think?
Thanks in advanced.
Hi I have an existing Kafka cluster with ZooKeeper cluster in EKS in AWS. (I built this cluster by using Strimzi operator and Kafka Kind). I failed to add this existing cluster to CMAK, after I successfully installed CMAK by cmak-operator chart. Is there anything wrong with my procedure?
kubectl -n kafka get pods
NAME READY STATUS RESTARTS AGE
cn-northwest-1-ops-kafka-crd-entity-operator-546c96d747-nggbz 3/3 Running 0 38s
cn-northwest-1-ops-kafka-crd-kafka-0 1/1 Running 0 2m41s
cn-northwest-1-ops-kafka-crd-kafka-1 1/1 Running 0 2m41s
cn-northwest-1-ops-kafka-crd-kafka-2 1/1 Running 0 2m41s
cn-northwest-1-ops-kafka-crd-zookeeper-0 1/1 Running 3 (3m44s ago) 5m45s
cn-northwest-1-ops-kafka-crd-zookeeper-1 1/1 Running 4 (3m4s ago) 5m45s
cn-northwest-1-ops-kafka-crd-zookeeper-2 1/1 Running 0 5m45s
strimzi-cluster-operator-5c8c984b7b-tcfnt 1/1 Running 0 4h22m
From the above screenshot, I added zookeeper hosts info into the second box.
cn-northwest-1-ops-kafka-crd-zookeeper-0:2181, cn-northwest-1-ops-kafka-crd-zookeeper-1:2181, cn-northwest-1-ops-kafka-crd-zookeeper-2:2181/kafka
Then, got timeout error:
With docker pull limits these days, it would be nice if the image could be pulled from an alternate repository provided in values.
Hello!
Due to zookeeper has no storage, and cluster config is applied only once on creation via job, cluster list is not persistent and will disappear after every CMAK pod recreation.
There is 2 possible ways to solve this:
We are using managed K8s installation in Google Cloud Platform (GKE) and chart installation fails with
Error: INSTALLATION FAILED: chart requires kubeVersion: >=1.21 which is incompatible with Kubernetes v1.21.5-gke.1302
Looking at the code, it's possible to parametrise Ingress
object, based on K8s version using Capabilities built-in object, so kubeVersion
could be dropped.
Roughly, this would look like the following:
templates/_utils.tpl
:
...
{{- define "kubeVersion" -}}
{{- default .Capabilities.KubeVersion.Version (regexFind "v[0-9]+\\.[0-9]+\\.[0-9]+" .Capabilities.KubeVersion.Version) -}}
{{- end -}}
{{- define "ingress.apiVersion" -}}
{{- if and (.Capabilities.APIVersions.Has "networking.k8s.io/v1") (semverCompare ">= 1.19.x" (include "kubeVersion" .)) -}}
{{- print "networking.k8s.io/v1" -}}
{{- else if .Capabilities.APIVersions.Has "networking.k8s.io/v1beta1" -}}
{{- print "networking.k8s.io/v1beta1" -}}
{{- else -}}
{{- print "extensions/v1beta1" -}}
{{- end -}}
{{- end -}}
templates/ingress.yaml
apiVersion: {{ template "ingress.apiVersion" . }}
...
That would make chart more versatile. If this approach is fine with you, I can open a PR.
Дякую / Thanks.
Is there an option to change the JMX port to a custom port? I use AWS MSK and the default JMX port is 1101. where can I change that?
Hi there,
I am trying to troubleshoot an issue where the Consumer data is missing when we add one particular cluster to the pool.
In the UI container logs I see
[ERROR] k.m.a.c.KafkaAdminClientActor - AdminClient not initialized yet, cannot process request : KAGetGroupSummary(ArrayBuffer(),[])
Once we remove this particular cluster from the pool of clusters, all data related to consumer for all clusters are visible.
Any help with debugging this issue is appreciated.
Version used: 2.2.0
i see you are deleting images from dockerhub. last week i was still able to install 1.3.0 but now images are deleted from there. can we set a longer retention period there? keeping last 2 majors can result in chart breaking every couple of weeks. this will not make this really available for production use....
In cases where global values are provided or if this chart is included as a subchart (even if no globals are actually specified), the values schema is overly restrictive and fails validation.
Hi CMAK team!
I'm implementing this operator right now and I'm having issues on activating LDAP. Here is the chart I'm using to configure that.
# various settings for CMAK UI container and CMAK application
ui:
port: 9000
# additional command line arguments
extraArgs:
- "-DbasicAuthentication.enabled=true"
- "-DbasicAuthentication.ldap.enabled=true"
- "-DbasicAuthentication.ldap.server=corpdsuat.company.com"
- "-DbasicAuthentication.ldap.port=389"
- "-DbasicAuthentication.ldap.username=uid=$capturedLogin$,ou=people,o=company.com,o=email"
- "-DbasicAuthentication.ldap.password='$capturedPassword'"
- "-DbasicAuthentication.ldap.search-base-dn=ou=people,o=company.com,o=email"
- "-DbasicAuthentication.ldap.connection-pool-size=10"
- "-DbasicAuthentication.ldap.ssl=false"
- "-DbasicAuthentication.ldap.ssl-trust-all=true"
- "-DbasicAuthentication.ldap.starttls=false"
With this code, when I get to the URL, it prompts me for user and password. Once I enter them I get the "Oops, an error occurred" and on the pod logs I get:
! @8243ilg5n - Internal server error, for (GET) [/] ->
play.api.UnexpectedException: Unexpected exception[LDAPBindException: invalid credentials]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:178)
at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:382)
at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:380)
at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
Caused by: com.unboundid.ldap.sdk.LDAPBindException: invalid credentials
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2273)
at com.unboundid.ldap.sdk.LDAPConnection.<init>(LDAPConnection.java:670)
at com.unboundid.ldap.sdk.LDAPConnection.<init>(LDAPConnection.java:563)
at controllers.LDAPAuthenticator.ldapConnectionPool$lzycompute(BasicAuthenticationFilter.scala:156)
at controllers.LDAPAuthenticator.ldapConnectionPool(BasicAuthenticationFilter.scala:144)
at controllers.LDAPAuthenticator.isAuthorized(BasicAuthenticationFilter.scala:195)
at controllers.LDAPAuthenticator.checkAuthentication(BasicAuthenticationFilter.scala:169)
at controllers.BasicAuthenticationFilter.apply(BasicAuthenticationFilter.scala:30)
at play.api.mvc.Filter$$anon$2.apply(Filters.scala:63)
at play.api.mvc.Filter$$anon$2.apply(Filters.scala:50)
2023-06-28 13:47:30,569 - [ERROR] application -
The credentials are not invalid. I have just changed and tested it on another application that uses the same LDAP server. That one is working fine.
Is there anything I have to change on the username and password configuration for the chart?
Thank you very much!
Helm style values for
First off all great work (=
I'm currently trying to debug the connection from cmake-operator to zookeeper.
Version: 1.4.2
First my values config looked like this:
21 cmak:
22 clusters:
23 - name: "stage"
24 kafkaVersion: "2.7.0"
25 curatorConfig:
26 zkConnect: "stage-zookeeper-client:2181"
For debugging I tried to add some extraArgs to the ui like:
21 cmak:
22 clusters:
23 - name: "stage"
24 kafkaVersion: "2.7.0"
25 curatorConfig:
26 zkConnect: "stage-zookeeper-client:2181"
27 ui:
28 extraArgs:
29 - "-Djavax.net.debug=ssl,handshake"
As I try to update the helm chart I get following message:
Helm upgrade failed: values don't meet the specifications of the schema(s) in the following chart(s):\ncmak-operator:\n- cmak: Additional property ui is not allowed
I'm trying to use this helm chart, but it seem that there is a problem:
# helm repo add cmak-operator https://eshepelyuk.github.io/cmak-operator/
Error: looks like "https://eshepelyuk.github.io/cmak-operator/" is not a valid chart repository or cannot be reached: failed to fetch https://eshepelyuk.github.io/cmak-operator/index.yaml : 404 Not Found
It's quite perplexing that I seem to be the only one experiencing this issue. I've made attempts to resolve it on multiple computers where I have Helm installed, yet the error persists consistently across all of them
Thanks in advance.
how can I pass array like basicAuthentication.excluded=["/cmak/api/health"] in ui.extraArgs ? seems everyting will convert to string
The latest version supports new versions of kafka
yahoo/CMAK#871
later deployments of the chart currently fail due to job not being able to update properly...
found this solution here
i will implement with proper values as soon as i can...
ii is my understanding that due to the limitations of cmak, which specifies that 1 consumer properties object will apply to all clusters effectively requiring allowing the same certificate across all managed clusters, is not the best solution. i do need this in my implementation.
so the question is, can we add this support here and allow passing consumer properties to the chart using it's values. if this is accepted i will add the support on my open fork and open the respective PR. WDYT?
I want to change the context path of the UI to work behind ingress.
is it possible to expose the "play.http.context" in values.yaml?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.