Giter VIP home page Giter VIP logo

Comments (12)

imalik8088 avatar imalik8088 commented on July 16, 2024 1

Yeah buddy thx very much for the help and once again great work! Would love to provide a more sophisticated example project, after my short holiday. Examining currently your Ansible module at work were the customer has some requirements or an old way to manage there topics with custom gradle, ruby and bash scripts šŸ¤• šŸ¤’ lets see if I can make world a better place to live šŸ˜„

from ansible-kafka-admin.

StephenSorriaux avatar StephenSorriaux commented on July 16, 2024

Hi @imalik8088, thanks for your message!

It seems like the kafka-python lib cannot be reached, did you pip install -r requirements.txt from inside a virtualenv? On your host, can you do a python -c 'from kafka.client import KafkaClient' without any problem?

from ansible-kafka-admin.

imalik8088 avatar imalik8088 commented on July 16, 2024

it seems that my laptop and python are not working correctly. The inline command with python -c was also not able to find the lib. I wish could understand python better šŸ’­

I have installed on my local machine python@2 via homebrew and my python --version is 2.7.5 have also tried todo pip3 install -r requirements.txt

from ansible-kafka-admin.

imalik8088 avatar imalik8088 commented on July 16, 2024

ok think I found the solution :-) python -m pip install -r requirements.txt for my mac and homebrew installtion. but now the next issue

The full traceback is:
Traceback (most recent call last):
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py", line 363, in <module>
    class CreatePartitionsRequest_v0(Request):
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py", line 366, in CreatePartitionsRequest_v0
    RESPONSE_TYPE = CreatePartitionsResponse_v0
NameError: name 'CreatePartitionsResponse_v0' is not defined

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py\", line 363, in <module>\n    class CreatePartitionsRequest_v0(Request):\n  File \"/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_KguwOs/ansible_module_kafka_lib.py\", line 366, in CreatePartitionsRequest_v0\n    RESPONSE_TYPE = CreatePartitionsResponse_v0\nNameError: name 'CreatePartitionsResponse_v0' is not defined\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}

https://github.com/StephenSorriaux/ansible-kafka-admin/blob/master/library/kafka_lib.py#L366 I think the issue is here. The CreatePartitionsResponse_v0 is not defined.

from ansible-kafka-admin.

StephenSorriaux avatar StephenSorriaux commented on July 16, 2024

Great. Are you using the 1.4.2 version of kafka-python? The CreatePartitionsResponse_v0 is taken from this library and imported via from kafka.protocol.admin import *.

from ansible-kafka-admin.

imalik8088 avatar imalik8088 commented on July 16, 2024

Wow thx for the support! But unfortunately it bumped again

The full traceback is:
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_aJu4GQ/ansible_module_kafka_lib.py", line 758, in main
    manager = KafkaManager(module=module, bootstrap_servers=bootstrap_servers, security_protocol=security_protocol, api_version=api_version, ssl_check_hostname=ssl_check_hostname, ssl_cafile=ssl_files['cafile']['path'], ssl_certfile=ssl_files['certfile']['path'], ssl_keyfile=ssl_files['keyfile']['path'], ssl_password=ssl_password, ssl_crlfile=ssl_files['crlfile']['path'], sasl_mechanism=sasl_mechanism, sasl_plain_username=sasl_plain_username, sasl_plain_password=sasl_plain_password, sasl_kerberos_service_name=sasl_kerberos_service_name)
  File "/var/folders/yk/h6_k8nrn4_bf50dv02_qkltr0000gn/T/ansible_aJu4GQ/ansible_module_kafka_lib.py", line 382, in __init__
    self.client = KafkaClient(**configs)
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/kafka/client_async.py", line 192, in __init__
    self.config['api_version'], str(self.API_VERSIONS)))

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "1.0.1",
            "bootstrap_servers": "localhost:9092",
            "name": "test",
            "options": {
                "flush.ms": 12345,
                "retention.ms": 574930
            },
            "partitions": 2,
            "replica_factor": 1,
            "resource": "topic",
            "sasl_kerberos_service_name": null,
            "sasl_mechanism": "PLAIN",
            "sasl_plain_password": null,
            "sasl_plain_username": null,
            "security_protocol": "PLAINTEXT",
            "ssl_cafile": null,
            "ssl_certfile": null,
            "ssl_check_hostname": true,
            "ssl_crlfile": null,
            "ssl_keyfile": null,
            "ssl_password": null,
            "state": "present",
            "zookeeper": "localhost:2181",
            "zookeeper_auth_scheme": "digest",
            "zookeeper_auth_value": ""
        }
    },
    "msg": "Error while initializing Kafka client : api_version [('1', '0', '1')] must be one of: [(0, 10, 1), (0, 10, 0), (0, 10), (0, 9), (0, 8, 2), (0, 8, 1), (0, 8, 0)] "
}

I'm using confluent-4.1.1, means Kafka 1.1.x https://docs.confluent.io/current/installation/versions-interoperability.html#cp-versions

from ansible-kafka-admin.

StephenSorriaux avatar StephenSorriaux commented on July 16, 2024

I think you are not using the 1.4.2 version of kafka-python since the api_version check was removed from the 1.4.2 version (dpkp/kafka-python@22e3f75)

from ansible-kafka-admin.

imalik8088 avatar imalik8088 commented on July 16, 2024

Had to reinstall the deps but i thinks it looks good. Now I have to solve somehow the connection issue.

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "1.0.1",
            "bootstrap_servers": "localhost:9092",
            "name": "test",
            "options": {
                "flush.ms": 12345,
                "retention.ms": 574930
            },
            "partitions": 2,
            "replica_factor": 1,
            "resource": "topic",
            "sasl_kerberos_service_name": null,
            "sasl_mechanism": "PLAIN",
            "sasl_plain_password": null,
            "sasl_plain_username": null,
            "security_protocol": "PLAINTEXT",
            "ssl_cafile": null,
            "ssl_certfile": null,
            "ssl_check_hostname": true,
            "ssl_crlfile": null,
            "ssl_keyfile": null,
            "ssl_password": null,
            "state": "present",
            "zookeeper": "localhost:2181",
            "zookeeper_auth_scheme": "digest",
            "zookeeper_auth_value": ""
        }
    },
    "msg": "Connection is not ready, please check your client and server configurations."
}

from ansible-kafka-admin.

StephenSorriaux avatar StephenSorriaux commented on July 16, 2024

OK, have you got any logs from your Kafka server? Can you give its configuration?

from ansible-kafka-admin.

imalik8088 avatar imalik8088 commented on July 16, 2024

Strage when I run the playbook they no logs from from Kafka or ZK are shown up. Just for testing purpose I have used curl with localhost:2181 and the ZK logs appeared šŸ¤”

I have upload now the server and the Ansible project:

I'm running the Kafka Cluster with with docker. If you want to try it out. Just clone the following repo https://github.com/imalik8088/kafka-on-demand-docker and docker-compose -f my-stack.yml up

And the Ansible stuff is here: https://github.com/imalik8088/ansible-kafka-topics-example and there I have run ansible-playbook deploy-topics.yml

from ansible-kafka-admin.

StephenSorriaux avatar StephenSorriaux commented on July 16, 2024

Thanks for the code, it was very helpful for debugging.

The problem comes from the Kafka broker configuration, it currently is advertising to PLAINTEXT://kafka:9092 which is not reachable from outside the container unless you add an entry to your /etc/hosts file with the current IP of the Kafka container. This is not very easy to use since if the container's IP change, you will need to update your /etc/hosts... To avoid this, here the configuration of your Kafka broker for the my-stack.yml file:

  kafka:
    image: confluentinc/cp-kafka:${CONFLUENT_VERSION}
    container_name: kafka_kodd
    hostname: kafka
    depends_on:
      - zookeeper
    ports:
      - "9094:9094"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092,OUTSIDE://localhost:9094"
      KAFKA_LISTENERS: "PLAINTEXT://:9092,OUTSIDE://:9094"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_LOG4J_LOGGERS: "kafka.controller=DEBUG,kafka.producer.async.DefaultEventHandler=DEBUG,state.change.logger=DEBUG"
      KAFKA_LOG4J_ROOT_LOGLEVEL: DEBUG
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DELETE_TOPIC_ENABLE: "true"

With this configuration, your Kafka broker will be reachable from outside the container via PLAINTEXT://localhost:9094. Change the port accordingly inside your tasks/main.yml and please use the master branch of my library. Then you will be good to go:
(ansible)

$ ansible-playbook deploy-topics.yml -vv
ansible-playbook 2.5.2
  config file = /home/stephen/dev/tmp/ansible-kafka-topics-example/ansible.cfg
  configured module search path = [u'/home/stephen/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]
Using /home/stephen/dev/tmp/ansible-kafka-topics-example/ansible.cfg as config file
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match
'all'


PLAYBOOK: deploy-topics.yml *******************************************************************************************
1 plays in deploy-topics.yml

PLAY [127.0.0.1] ******************************************************************************************************
META: ran handlers

TASK [kafka_lib : create topic] ***************************************************************************************
task path: /home/stephen/dev/tmp/ansible-kafka-topics-example/roles/kafka_lib/tasks/main.yml:1
changed: [127.0.0.1] => {"changed": true, "msg": "topic 'test1': successfully created."}
META: ran handlers
META: ran handlers

PLAY RECAP ************************************************************************************************************
127.0.0.1                  : ok=1    changed=1    unreachable=0    failed=0

(kafka)

root@kafka:/# kafka-topics --describe --zookeeper zookeeper:2181
Topic:__confluent.support.metrics	PartitionCount:1	ReplicationFactor:1	Configs:retention.ms=31536000000
	Topic: __confluent.support.metrics	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
Topic:_schemas	PartitionCount:1	ReplicationFactor:1	Configs:cleanup.policy=compact
	Topic: _schemas	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
Topic:test1	PartitionCount:2	ReplicationFactor:1	Configs:retention.ms=574930,flush.ms=12345
	Topic: test1	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: test1	Partition: 1	Leader: 1	Replicas: 1	Isr: 1

from ansible-kafka-admin.

StephenSorriaux avatar StephenSorriaux commented on July 16, 2024

Sure, Iā€™m happy to help. Feel free to submit any bugs or new features in the future.

Iā€™m now closing this issue since the underlying problem is now solved.

from ansible-kafka-admin.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    šŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. šŸ“ŠšŸ“ˆšŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ā¤ļø Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.