Comments (12)
Yeah buddy thx very much for the help and once again great work! Would love to provide a more sophisticated example project, after my short holiday. Examining currently your Ansible module at work were the customer has some requirements or an old way to manage there topics with custom gradle, ruby and bash scripts š¤ š¤ lets see if I can make world a better place to live š
from ansible-kafka-admin.
Hi @imalik8088, thanks for your message!
It seems like the kafka-python
lib cannot be reached, did you pip install -r requirements.txt
from inside a virtualenv? On your host, can you do a python -c 'from kafka.client import KafkaClient'
without any problem?
from ansible-kafka-admin.
it seems that my laptop and python are not working correctly. The inline command with python -c
was also not able to find the lib. I wish could understand python better š
I have installed on my local machine python@2 via homebrew and my python --version
is 2.7.5
have also tried todo pip3 install -r requirements.txt
from ansible-kafka-admin.
ok think I found the solution :-) python -m pip install -r requirements.txt
for my mac and homebrew installtion. but now the next issue
The full traceback is:
Traceback (most recent call last):
File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py", line 363, in <module>
class CreatePartitionsRequest_v0(Request):
File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py", line 366, in CreatePartitionsRequest_v0
RESPONSE_TYPE = CreatePartitionsResponse_v0
NameError: name 'CreatePartitionsResponse_v0' is not defined
fatal: [127.0.0.1]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py\", line 363, in <module>\n class CreatePartitionsRequest_v0(Request):\n File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py\", line 366, in CreatePartitionsRequest_v0\n RESPONSE_TYPE = CreatePartitionsResponse_v0\nNameError: name 'CreatePartitionsResponse_v0' is not defined\n",
"module_stdout": "",
"msg": "MODULE FAILURE",
"rc": 1
}
https://github.com/StephenSorriaux/ansible-kafka-admin/blob/master/library/kafka_lib.py#L366 I think the issue is here. The CreatePartitionsResponse_v0
is not defined.
from ansible-kafka-admin.
Great. Are you using the 1.4.2
version of kafka-python
? The CreatePartitionsResponse_v0
is taken from this library and imported via from kafka.protocol.admin import *
.
from ansible-kafka-admin.
Wow thx for the support! But unfortunately it bumped again
The full traceback is:
File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_aJu4GQ/ansible_module_kafka_lib.py", line 758, in main
manager = KafkaManager(module=module, bootstrap_servers=bootstrap_servers, security_protocol=security_protocol, api_version=api_version, ssl_check_hostname=ssl_check_hostname, ssl_cafile=ssl_files['cafile']['path'], ssl_certfile=ssl_files['certfile']['path'], ssl_keyfile=ssl_files['keyfile']['path'], ssl_password=ssl_password, ssl_crlfile=ssl_files['crlfile']['path'], sasl_mechanism=sasl_mechanism, sasl_plain_username=sasl_plain_username, sasl_plain_password=sasl_plain_password, sasl_kerberos_service_name=sasl_kerberos_service_name)
File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_aJu4GQ/ansible_module_kafka_lib.py", line 382, in __init__
self.client = KafkaClient(**configs)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/kafka/client_async.py", line 192, in __init__
self.config['api_version'], str(self.API_VERSIONS)))
fatal: [127.0.0.1]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"api_version": "1.0.1",
"bootstrap_servers": "localhost:9092",
"name": "test",
"options": {
"flush.ms": 12345,
"retention.ms": 574930
},
"partitions": 2,
"replica_factor": 1,
"resource": "topic",
"sasl_kerberos_service_name": null,
"sasl_mechanism": "PLAIN",
"sasl_plain_password": null,
"sasl_plain_username": null,
"security_protocol": "PLAINTEXT",
"ssl_cafile": null,
"ssl_certfile": null,
"ssl_check_hostname": true,
"ssl_crlfile": null,
"ssl_keyfile": null,
"ssl_password": null,
"state": "present",
"zookeeper": "localhost:2181",
"zookeeper_auth_scheme": "digest",
"zookeeper_auth_value": ""
}
},
"msg": "Error while initializing Kafka client : api_version [('1', '0', '1')] must be one of: [(0, 10, 1), (0, 10, 0), (0, 10), (0, 9), (0, 8, 2), (0, 8, 1), (0, 8, 0)] "
}
I'm using confluent-4.1.1, means Kafka 1.1.x https://docs.confluent.io/current/installation/versions-interoperability.html#cp-versions
from ansible-kafka-admin.
I think you are not using the 1.4.2 version of kafka-python
since the api_version
check was removed from the 1.4.2 version (dpkp/kafka-python@22e3f75)
from ansible-kafka-admin.
Had to reinstall the deps but i thinks it looks good. Now I have to solve somehow the connection issue.
fatal: [127.0.0.1]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"api_version": "1.0.1",
"bootstrap_servers": "localhost:9092",
"name": "test",
"options": {
"flush.ms": 12345,
"retention.ms": 574930
},
"partitions": 2,
"replica_factor": 1,
"resource": "topic",
"sasl_kerberos_service_name": null,
"sasl_mechanism": "PLAIN",
"sasl_plain_password": null,
"sasl_plain_username": null,
"security_protocol": "PLAINTEXT",
"ssl_cafile": null,
"ssl_certfile": null,
"ssl_check_hostname": true,
"ssl_crlfile": null,
"ssl_keyfile": null,
"ssl_password": null,
"state": "present",
"zookeeper": "localhost:2181",
"zookeeper_auth_scheme": "digest",
"zookeeper_auth_value": ""
}
},
"msg": "Connection is not ready, please check your client and server configurations."
}
from ansible-kafka-admin.
OK, have you got any logs from your Kafka server? Can you give its configuration?
from ansible-kafka-admin.
Strage when I run the playbook they no logs from from Kafka or ZK are shown up. Just for testing purpose I have used curl
with localhost:2181 and the ZK logs appeared š¤
I have upload now the server and the Ansible project:
I'm running the Kafka Cluster with with docker. If you want to try it out. Just clone the following repo https://github.com/imalik8088/kafka-on-demand-docker and docker-compose -f my-stack.yml up
And the Ansible stuff is here: https://github.com/imalik8088/ansible-kafka-topics-example and there I have run ansible-playbook deploy-topics.yml
from ansible-kafka-admin.
Thanks for the code, it was very helpful for debugging.
The problem comes from the Kafka broker configuration, it currently is advertising to PLAINTEXT://kafka:9092
which is not reachable from outside the container unless you add an entry to your /etc/hosts
file with the current IP of the Kafka container. This is not very easy to use since if the container's IP change, you will need to update your /etc/hosts
... To avoid this, here the configuration of your Kafka broker for the my-stack.yml
file:
kafka:
image: confluentinc/cp-kafka:${CONFLUENT_VERSION}
container_name: kafka_kodd
hostname: kafka
depends_on:
- zookeeper
ports:
- "9094:9094"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092,OUTSIDE://localhost:9094"
KAFKA_LISTENERS: "PLAINTEXT://:9092,OUTSIDE://:9094"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_LOG4J_LOGGERS: "kafka.controller=DEBUG,kafka.producer.async.DefaultEventHandler=DEBUG,state.change.logger=DEBUG"
KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_DELETE_TOPIC_ENABLE: "true"
With this configuration, your Kafka broker will be reachable from outside the container via PLAINTEXT://localhost:9094
. Change the port accordingly inside your tasks/main.yml
and please use the master branch of my library. Then you will be good to go:
(ansible)
$ ansible-playbook deploy-topics.yml -vv
ansible-playbook 2.5.2
config file = /home/stephen/dev/tmp/ansible-kafka-topics-example/ansible.cfg
configured module search path = [u'/home/stephen/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]
Using /home/stephen/dev/tmp/ansible-kafka-topics-example/ansible.cfg as config file
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match
'all'
PLAYBOOK: deploy-topics.yml *******************************************************************************************
1 plays in deploy-topics.yml
PLAY [127.0.0.1] ******************************************************************************************************
META: ran handlers
TASK [kafka_lib : create topic] ***************************************************************************************
task path: /home/stephen/dev/tmp/ansible-kafka-topics-example/roles/kafka_lib/tasks/main.yml:1
changed: [127.0.0.1] => {"changed": true, "msg": "topic 'test1': successfully created."}
META: ran handlers
META: ran handlers
PLAY RECAP ************************************************************************************************************
127.0.0.1 : ok=1 changed=1 unreachable=0 failed=0
(kafka)
root@kafka:/# kafka-topics --describe --zookeeper zookeeper:2181
Topic:__confluent.support.metrics PartitionCount:1 ReplicationFactor:1 Configs:retention.ms=31536000000
Topic: __confluent.support.metrics Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic:_schemas PartitionCount:1 ReplicationFactor:1 Configs:cleanup.policy=compact
Topic: _schemas Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic:test1 PartitionCount:2 ReplicationFactor:1 Configs:retention.ms=574930,flush.ms=12345
Topic: test1 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: test1 Partition: 1 Leader: 1 Replicas: 1 Isr: 1
from ansible-kafka-admin.
Sure, Iām happy to help. Feel free to submit any bugs or new features in the future.
Iām now closing this issue since the underlying problem is now solved.
from ansible-kafka-admin.
Related Issues (20)
- [Module] Add `kafka_users` module
- [Module] Unable to use ssl one way with zookeeper
- Output variable name difference for lib kafka_info is not optimal HOT 2
- UnboundLocalError: local variable 'manager' referenced before assignment while using SSL HOT 1
- Get unspecified Error Message on module kafka_info for topics HOT 3
- Unnecessary requirement for acl_operation in module kafka_acls HOT 1
- Unable to get Topic Information when connecting to a Cluster using SSL HOT 20
- spelling: Seomthing -> Something
- Question: How to add ACLs with transactional-ids? HOT 5
- kafka_topic: invalid check_mode behaviour when updating an existing topic HOT 2
- Enhancement: allow to define common configurations from env vars HOT 1
- Could not find imported module support code for kafka_users: Looked for ansible.module_utils.kafka_scram HOT 1
- kafka_user: Multiple mechanisms for password during user creation HOT 1
- Ansible-galaxy is not updated HOT 1
- Add publishing to Ansible Galaxy as collection HOT 4
- Missing asset in release 0.18.1 HOT 3
- Add quotas management for KRaft-based clusters HOT 2
- Consumer group ACL creation issue on 0.19.0 HOT 1
- outdated example HOT 1
- Add mTLS SSL support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ansible-kafka-admin.