Giter VIP home page Giter VIP logo

python-kafka-logging's Introduction

Python Kafka Logging Handler

Downloads Latest Version License

Simple python logging handler for forwarding logs to a kafka server.

The handler uses a logstash formatter.

Installing

$ pip install python_kafka_logging

How to use

Example for using the handler within a native logging.conf file

example/logging.conf

Get the project

  1. Clone the git repository
$ git clone https://github.com/taykey/python-kafka-logging/
  1. Install a virtualenv
$ sudo apt-get install python-virtualenv
  1. Create a new virtualenv
$ virtualenv python_kafka_logging_ve
  1. Install project's requirements
$ python_kafka_logging_ve/bin/pip install -r requirements.txt

Reporting Issues

If you have suggestions, bugs or other issues specific to this library, file them here or contact me at avihoo <at> taykey <dot> com.

keywords: python, logging, handler, example, kafka, logs, logstash, formatter

python-kafka-logging's People

Contributors

avihoo avatar coolbho3k avatar nivm avatar wayhome avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-kafka-logging's Issues

pip install error: ModuleNotFoundError: No module named 'pip.req'

when I use pip install python-kafka-logging, it alway get error like:
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
File "", line 1, in
File "C:\Local\Temp\pip-install-wopvdc79\python-kafka-logging\setup.py", line 6, in
from pip.req import parse_requirements
ModuleNotFoundError: No module named 'pip.req'
----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in C:\Local\Temp\pip-install-wopvdc79\python-kafka-logging
I tried to update my pip, but no use.
Finally, i download the projoect, and change the 'setup.py' in line 6:
from "from pip.req import parse_requirements" to:
"try:
from pip._internal.req import parse_requirements
except ImportError:
from pip.req import parse_requirements
"
That can compatible with different pip versions.

Suggest to allow passing any possible argument to the producer.

`
from kafka.client import KafkaClient
from kafka.producer import SimpleProducer, KeyedProducer
import logging
from kafka.conn import DEFAULT_SOCKET_TIMEOUT_SECONDS

class KafkaHandler(logging.Handler):

def __init__(self, hosts_list, topic, timeout_secs=DEFAULT_SOCKET_TIMEOUT_SECONDS, **kwargs):
    logging.Handler.__init__(self)

    self.kafka_client = KafkaClient(hosts_list, timeout=timeout_secs)
    self.key = kwargs.get("key", None)
    self.kafka_topic_name = topic

    if not self.key:
        self.producer = SimpleProducer(self.kafka_client, **kwargs)
    else:
        self.producer = KeyedProducer(self.kafka_client, **kwargs)

def emit(self, record):
    # drop kafka logging to avoid infinite recursion
    if record.name == 'kafka':
        return
    try:
        # use default formatting
        msg = self.format(record)
        if isinstance(msg, unicode):
            msg = msg.encode("utf-8")

        # produce message
        if not self.key:
            self.producer.send_messages(self.kafka_topic_name, msg)
        else:
            self.producer.send_messages(self.kafka_topic_name, self.key, msg)
    except (KeyboardInterrupt, SystemExit):
        raise
    except:
        self.handleError(record)

def close(self):
    if self.producer is not None:
        self.producer.stop()
    logging.Handler.close(self)

`

Could not install the package using pip 1.3.1

error:
Running setup.py egg_info for package python-kafka-logging
Traceback (most recent call last):
File "", line 16, in
File "/tmp/pip-build-root/python-kafka-logging/setup.py", line 23, in
install_requirements = parse_requirements('requirements.txt', None, None, None, uuid.uuid1())
TypeError: parse_requirements() takes at most 4 arguments (5 given)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

AttributeError: 'KafkaLoggingHandler' object has no attribute 'producer'

Hello
I use the check_kafka script and I have the following error message, and I do not see where the problem is, below the logging.conf file

[loggers]
keys=root

[handlers]
keys=consoleHandler,fileHandler,kafkaHandler

[formatters]
keys=simpleFormatter,logstashFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler,fileHandler,kafkaHandler
formatter=simpleFormatter

[handler_fileHandler]
class=logging.handlers.WatchedFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/check_kafka.log', 'a')

[handler_kafkaHandler]
class=python_kafka_logging.KafkaHandler.KafkaLoggingHandler
level=DEBUG
formatter=logstashFormatter
args=("192.168.0.35:2181, 192.168.0.193:2181","apps-openshift")

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(levelname)s - %(name)s - %(message)s

[formatter_logstashFormatter]
class=logstash_formatter.LogstashFormatter
format={"extra": {"appName": "myPythonApp", "environment": "AWS-Test"}}

can you help me ?
thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.