Comments (33)
Thanks. Seems to work now with the following env vars on the Deployment spec:
containers:
- name: my-cluster-schema-registry
image: confluentinc/cp-schema-registry:4.1.1-2
ports:
- containerPort: 8081
env:
- name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
value: PLAINTEXT://my-cluster-kafka:9092
- name: SCHEMA_REGISTRY_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SCHEMA_REGISTRY_LISTENERS
value: http://0.0.0.0:8081
- name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL
value: PLAINTEXT
from strimzi-kafka-operator.
My view which might not be necessarily the view of other maintainers:
- There are some recognized open source linceses such as Apache Lincese 2.0 etc. These are well known licenses with well known working combinations. Once you have a propriatery license (although open source) such as the Confluent Community License, it is always up to the layers to decide what is and isn't compatible. It is not necessarily about Strimzi violating the Confluent lincese - but about the Confluent license violating our Apache Lincese 2.0.
- Another aspect is that we are not Confluent. And while it is Confluent's right to block SaaS companies from using their software, I want Strimzi to be a project open to and usable for everyone including SaaS companies. So I do want the things which we integrate into Strimzi to be usable even by them. Which the Confluent Schema Registry doesn't fulfil.
from strimzi-kafka-operator.
Redhat developed a confluent schema registry compatible schema registry, maybe it can be integrated?
from strimzi-kafka-operator.
I can understand why Strimzi does not have its own Registry now.
Because there are many popular solutions there especially Apicurio Registry and Confluent Schema Registry. And they are working with Kafka cluster created by Strimzi directly.
I listed all info I found at here hongbo-miao/hongbomiao.com#8348
And I have succeed both Apicurio Registry and Confluent Schema Registry in a Kafka cluster created by Strimzi. Hopefully that ticket helps more people. 😃
from strimzi-kafka-operator.
If someone still interested, i have my supporeted helm chart for apicurio and apicurio content sync.
https://github.com/eshepelyuk/apicurio-registry-helm
Not tried with Strimzi, but feel free to open issues and pull requests.
from strimzi-kafka-operator.
I don't think we are planning anything around service registry. Sorry.
from strimzi-kafka-operator.
TBH, I actually haven't seen this before. So I will need to have a bit closer look.
from strimzi-kafka-operator.
@scholzj any way to sponsor adding support for Kafka registry?
from strimzi-kafka-operator.
Could someone provide an example how to run cp-schema-registry-server with strimzi? Especially if authentication is plain and the connection part to zookeeper. @scholzj mentioned this here, is this the way to go for the zookeeper part?
from strimzi-kafka-operator.
@scholzj Actually apicurio works pretty well with Strimzi. As you said it is just another kafka-client. Out of the box it does not support Oauth2, so i added that functionality. Basically it just means adding some dependencies.
I put together a docker-build script that does this.
https://github.com/iceman91176/witcom-apicurio-registry
This is pretty much undocumented right now ;-)
I also have a helm-chart available that deploys the registry and configures oauth2. This one is not online yet, if anyone is interested i will share it.
Currently apicurio does not support any authentication/authorization. They will have it in Version 2.0 - based on keycloak. Until then there is a "security-gateway" available that provides role-based access to the schema-registry.
https://github.com/witcom-gmbh/apicurio-security-gateway
This (plus securing the Registry-UI) is also integrated in the helm-chart i mentioned above. I should be able to provide that in the next days. I also have a avro-producer/consumer demo available that uses apicurio natively.
from strimzi-kafka-operator.
This would be appreciated. I tried setting up a Kub Service & Deployment of the Schema Registry using the confluentinc/cp-schema-registry:4.1.1-2 image with bootstrap servers (vs ZK).
I got past complaints about PLAINTEXT not matching broker setup by adding:
- name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL
value: SSL
currently having problem with repeated versions of this:
[kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
[kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.common.network.SslTransportLayer - Failed to send SSL Close message
from strimzi-kafka-operator.
As of now we don't use SSL - only PLANTEXT. So I think this is where the problem is coming from. There is a PR for SSL support, it should land in master soon (#487 ), that might help once its merged.
from strimzi-kafka-operator.
This doesn't seem to be working for me. I am also trying to get a schema-registry instance to run in my kafka cluster. I have a connect image from the strimzi source that has the debezium mysql plugin and the confluent avro converter installed. I have a deployment set up for the schema-registry. If I don't specify the SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL
env variable it complains about the SCHEMA_REGISTRY_PORT
not being set, and if I set the PORT it tells me to use the URL env variable. None of that is the concern of the good people developing the strimzi-kafka-operator.
I saw in this issue that a Network policy is necessary to connect an outside deployment. I added this network policy and after that I was at least getting through to the zookeeper service, but now I see that the tis-sidecar is complaining about an SSL version?
2019.03.15 22:13:51 LOG3[1:140278096041728]: SSL_accept: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2019.03.15 22:13:51 LOG5[1:140278096041728]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
I'm not sure if that message is a red herring or if there's a setting I'm missing.
from strimzi-kafka-operator.
I ended up solving the above by changing the name of my Deployment and Service in kubernetes. Related to this issue: confluentinc/schema-registry#689
from strimzi-kafka-operator.
@jslusher glad to see that now it's working for you. I am going to close this issue, feel free to reopen if you have additional questions/problems.
from strimzi-kafka-operator.
I'm encountering the same issue as jslusher, and changing the names did not help. Here is my deployment:
---
apiVersion: v1
kind: Service
metadata:
name: registry-client
namespace: kafka
spec:
ports:
- port: 8081
clusterIP: None
selector:
app: my-registry
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: kafka
spec:
selector:
matchLabels:
app: "my-registry"
replicas: 1
template:
metadata:
labels:
app: my-registry
spec:
terminationGracePeriodSeconds: 10
containers:
- name: cp-registry-container
image: confluentinc/cp-schema-registry
env: # see https://docs.confluent.io/current/schema-registry/installation/config.html
- name: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL
value: my-cluster-zookeeper-client:2181
- name: SCHEMA_REGISTRY_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SCHEMA_REGISTRY_AVRO_COMPATIBILITY_LEVEL
value: BACKWARD
ports:
- containerPort: 8081
The schema_registry pod is receiving the following error:
java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
And the zookeeper pod is receiving this error:
SSL_accept: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
I was looking through other issues and noticed SSL was recently added to strimzi, so I wasn't sure whether I need to use the - name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL value: PLAINTEXT
variable, as mentioned above. I tried this anyway and it didn't seem to resolve the issue.
Thank you!
from strimzi-kafka-operator.
I have no experience with Confluent Schema Registry. But you point the variable SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL
to Zookeeper. Is that correct? Or should it point to Kafka? I think that in the initial question in this issue it was pointing to Kafka.
In general, our Zookeeper is locked down for security reasons using TLS. If you need to access it directly, you should be able to use this (on your own risk).
from strimzi-kafka-operator.
Thanks, I've been working through this issue, still haven't found a solution, but I noticed that on the project board, there is a todo item for this feature, do you plan on implementing this in the foreseeable future?
Thanks!
from strimzi-kafka-operator.
We do plan it, but I'm afraid I do not have any exact timeline.
from strimzi-kafka-operator.
Changing the configuration to these settings fixed my issues: #29 (comment)
from strimzi-kafka-operator.
I do not think there is any other way then by opening PRs. We are looking into the Schema Registry, but it is not trivial also given we cannot use the Confluent one due to the license.
That said, I think some people got the Confluent chema registry working as any other client connecting to Kafka. IIRC the setup is not completely straight forward, but it can be configure to not use Zookeeper and connect to Kafka only. (In which case, if you deploy it as your own application, the license is really only about your use cases. Where we as a project want tosmething what everyone can use.)
from strimzi-kafka-operator.
@scholzj thank you for replying.
I do not think there is any other way then by opening PRs. We are looking into the Schema Registry, but it is not trivial also given we cannot use the Confluent one due to the license
Unfortunately, my DevOps skills are limited, that's why I asked if there is any way of supporting it.
we cannot use the Confluent one due to the license.
What's the issue with the Confluent's Community License?
Under the Confluent Community License, you can access the source code and modify or redistribute it; there is only one thing you cannot do, and that is use it to make a competing SaaS offering. Here is the exact language:
“Excluded Purpose” is making available any software-as a-service, platform-as-a-service, infrastructure-as-a-service or other similar online service that competes with Confluent products or services that provide the Software.
For example, it does not allow hosting of Confluent KSQL, Confluent Schema Registry, Confluent REST Proxy, or other software licensed under the Confluent Community License as online service offerings that compete with Confluent SaaS products or services that provide the same software. If you are not doing what is excluded, this license change will not affect you.
I'm not a lawyer but, once strimzi is an open-source initiative, I can't see how it violates the license.
from strimzi-kafka-operator.
I completely understand your point. However, using a schema registry is almost mandatory when using Avro, so maintaining it under the Strimzi stack is a significant benefit from a dev point of view.
from strimzi-kafka-operator.
I agree. But we need to find solution whcih works for everyone.
from strimzi-kafka-operator.
Is there any open source alternative to Schema Registry with a more permissive license, or anything currently in development?
from strimzi-kafka-operator.
Confluent’s schema registry is the
most popular implementation and in active development. The other option is https://github.com/schema-repo/schema-repo, but it looks abandoned.
from strimzi-kafka-operator.
@scholzj and @alexwennerberg just found this:
https://github.com/hortonworks/registry
from strimzi-kafka-operator.
@duanshiqiang The Apicurio Schema Registry should work without any problems with Strimzi. I would love to have some blog post or a better demo about how to integrate it. But I didn't had yet time for it. If anyone with more time would be interested in doing it, we would be more than happy to publish it of course.
However, I do not think we plan at this point any deeper integration into the operator. The Schema registry is more than happy to run as a separate deployment - it is basically just another application connecting to Kafka as a client. So I do not think there is much value in just creating the deployment by the operator. Do you have any ideas how could it be more integrated into the operator htan just creating the deployment?
@dklesev As for the Confluent Registry, TBH I'm not sure how much it needs the Zookeeper access - I think someone once told me that you can really configure it without the Zookeeper. But I personally haven't done it, so I'm not sure. Our Zookeeper is by default locked down using the TLS sidecars with TLS Client Authentication to make sure it is secure. The Gist you linked will open it to anyone without any authentication. That should make it possible to use it from any other applications, but it also means increased security risk and any badly written application using the Zookeeper can affect your Kafka cluster. So use at your own risk.
from strimzi-kafka-operator.
@scholzj what do you think about this, some parts are hardcoded, however. If I find time I will look into this and integration with strimzi. I think its quite important to have the schema registry (cp/rh) being easy to setup with strimzi as its a common component used with kafka.
from strimzi-kafka-operator.
I've open another java project to support schema registry operator https://github.com/shangyuantech/strimzi-registry-ksql-operator . Now it only provide some basic non SSL functions.
I would like to integrate the two services, and then adapt to the current Kafka Operator.
from strimzi-kafka-operator.
@iceman91176 please do share you helm chart
from strimzi-kafka-operator.
@scholzj Actually apicurio works pretty well with Strimzi. As you said it is just another kafka-client. Out of the box it does not support Oauth2, so i added that functionality. Basically it just means adding some dependencies.
I put together a docker-build script that does this.
https://github.com/iceman91176/witcom-apicurio-registry
This is pretty much undocumented right now ;-)I also have a helm-chart available that deploys the registry and configures oauth2. This one is not online yet, if anyone is interested i will share it.
Currently apicurio does not support any authentication/authorization. They will have it in Version 2.0 - based on keycloak. Until then there is a "security-gateway" available that provides role-based access to the schema-registry.
https://github.com/witcom-gmbh/apicurio-security-gateway
This (plus securing the Registry-UI) is also integrated in the helm-chart i mentioned above. I should be able to provide that in the next days. I also have a avro-producer/consumer demo available that uses apicurio natively.
Please share the helm-chart I will deploy in our Strimzi Kafka cluster.
from strimzi-kafka-operator.
I think this issue should be reopened
from strimzi-kafka-operator.
Related Issues (20)
- [ST] Add migration system tests with JBOD support in KRaft mode HOT 2
- [Bug]: the k8s apiserver using--goaway-chance flag resulted in createOrUpdate failed HOT 2
- [Bug]: Kafka/ZooKeeper cannot schedule due to pod antiaffinity HOT 3
- [Bug]: generateCertificatesSecret will exceed Secret max size HOT 2
- Performance Issue with SCRAM-SHA Credentials Cache at 4800+ `KafkaUsers` HOT 5
- [Enhancement]: Extend Kafka CRD to configure the jmxOptions for the remaining workloads HOT 5
- JMX with enabled authentication does not work on FIPS enabled Kubernetes clusters
- [Bug]: local keystore passwords appear in the process table HOT 5
- Remove KRaft JBOD storge check once support for Kafka 3.6.0 is dropped
- MM2 connector auto-restarting does not seem to work
- [Enhancement]: Allow configmaps/secrets to be used for javaSystemProperties HOT 3
- [Enhancement]: Improve tools/report.sh to make only the namespace required (make the cluster name optional) HOT 1
- [Bug]: TopicOperator removes throttled.replicas topic configuration of throttled reassignment/rebalance HOT 15
- [Docs]: Add compatibility matrix table to docs HOT 4
- [ST] MigrationST - some `KafkaTopics` are not deleted correctly after the migration tests
- [Bug]: KafkaUser secret decoding improperly HOT 2
- [Bug]: Current offset is higher than last offset
- [Bug]: Downtime due to pod readinessProbe when rolling update Kafka pod
- Logging configuration is never updated for Connect when connector operator is disabled HOT 1
- [Enhancement]: hot reload Kafka on changes in `brokerCertChainAndKey` instead of a rolling update HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from strimzi-kafka-operator.