cipriantarta / django-logging Goto Github PK

logs are arriving in kibana, all is working! thanks again for helping me out earlier.

Still two probably simple to cover questions on the code.

I have celery tasks running, related to users. these turn into results.

I now wanted to add results and user details also in kibana, including exceptions (to prevent time-out) when things go bad inside my taks.

I tried the: log.info('info to log') and it puts the "info to log" inside the "raw message" inside kibana.

My two questions:

• If I wanted an output that returns a "structured" by kibana interpretable output, so that the message.raw is transformed in message.request.id, message.request.name, etc. How do I do that in an easy way? Or do I manually format my log.info()-input in json-style?

• For exception handling, i noticed your "ErrorLogObject"-class. How does it work? As I am in the celery task, and requests are not serialisable, it is impossible to pass it on, does it mean I will not be able to use this here?

EDIT:
I figured out how ErrorLogObject works in views.py:

• request is needed - so my question is still valid for the case of a celery task
• I used except Exception as e: and used de "e" as exception input
• I set duration as an integer being "zero" which allows me to push error notifications to kibana.

EDIT 2:
Tried minimalistic json output in the log.error, something like:
log.error('{{ \'raw\' : {{ \'x\':{0}, \'y\':{1} }} }}'.format( x , y ) )
However, not splitting the "message.raw" into "message.x" and "message.y"

We are using the lib behind a proxy to connect to the elasticsearch. As the lib does not use the Requests transport, the program does not use the envvars HTTP_PROXY nor HTTPS_PROXY, and thus the connection fails.

Is there any way to set up django-logging so it creates the Elasticsearch with requests transport option enabled?

Thanks for the amazing work!

I receive the following error when starting the development server after adding django logging to installed apps and to the middleware section of the settings.py file.

Environment
Windows 10
Python 3.5.2
Django 1.10.2

Unhandled exception in thread started by <function check_errors..wrapper at 0x000001AF6A17ED08>
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\utils\autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\core\management\commands\runserver.py", line 142, in inner_run
handler = self.get_handler(*args, **options)
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\contrib\staticfiles\management\commands\runserver.py", line 27, in get_handler
handler = super(Command, self).get_handler(*args, **options)
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\core\management\commands\runserver.py", line 64, in get_handler
return get_internal_wsgi_application()
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\core\servers\basehttp.py", line 49, in get_internal_wsgi_application
return import_string(app_path)
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\utils\module_loading.py", line 20, in import_string
module = import_module(module_path)
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\importlib_init_.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "C:\dev\django\classregistration\classregistration\wsgi.py", line 17, in

application = get_wsgi_application()
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\core\wsgi.py", line 14, in get_wsgi_application
return WSGIHandler()
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\core\handlers\wsgi.py", line 153, in init
self.load_middleware()
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\django\core\handlers\base.py", line 82, in load_middleware

cursor_wrapper.py line 36:

Thread(target=do_log, args=(self.cursor, *args)).start()

is *args support python 2.7?

After I follow the installation I got ModuleNotFoundError: No module named 'django_logging'
Software:
Python3.6.2
Django1.11

Is it support Django 1.11 and Python3.6.2?

Hi Guys,

I was super excited to notice this json-logging-with-direct-eleastic-search-integration add-on for django, no need for Logstash or Fluentd > great!

However, I need some help in setting this up.

I have the installation working, I still have DISABLE_EXISTING_LOGGERS = False so I see the JSON logs appearing inside my command-line.

However, they don't get pushed toward my Elastic Cloud environment. I think this has to do with the way I am setting up the Host and AUTH inside the settings.py.

It would be great if somebody could help me out here.

here are my settings inside settings.py:

CONSOLE_LOG = True # Log to console.

SQL_LOG = True # Log SQL queries.

SQL_THRESHOLD = 0.5 # Log slow queries only.

LOG_LEVEL = 'debug' # If settings.DEBUG is set to True, otherwise LOG_LEVEL is set to 'info'

DISABLE_EXISTING_LOGGERS = False #True # Set this to False if you want to combine with multiple loggers.

LOG_PATH = '{}/logs'.format(BASE_DIR) # If the logs folder does not exist, it will be created.

IGNORED_PATHS = ['/admin', '/static', '/favicon.ico'] # List of URL endpoints to ignore.

RESPONSE_FIELDS = ('status', 'reason', 'charset', 'headers', 'content') # List of response fields to log.

CONTENT_JSON_ONLY = True # Log response content only if its a JSON document.

ROTATE_MB = 100 # Maximum size in MB that the log file can have before it gets rotated.

ROTATE_COUNT = 10 # Maximum number of rotated log files.

INDENT_CONSOLE_LOG = 2 # Indent console log by "n" spaces.

ELASTICSEARCH_ENABLED = True # Set to yes to enable elasticsearch support.

ELASTICSEARCH_HOSTS = ["https://e89046cda63602fdc974e90502252d6a.eu-west-1.aws.found.io:9243"] # Elasticsearch hosts

ELASTICSEARCH_INDEX = "django-logging-json" # Elasticsearch index name

ELASTICSEARCH_SSL = True # Elasticsearch connection via SSL (:443)

ELASTICSEARCH_AUTH = "elastic:xxxxxxx-my-password-xxxxxxx" 

Thanks, r

certifi gets imported, but not installed with the library.
This currently breaks the library for us 😞

So, python 3.9 changes some of the attributes of traceback.TracebackException which is being used here to get stack trace information in ErrorLogObject. Namely, the exc_traceback attribute has been removed. Fortunately, there is another way to get the same information, but the code as written right now will not work under python 3.9.

One possible example of how to re-write it for compatibility, though the code here removes support for python 3.4 and older: freedomofpress#6

I am experimenting with my #13
But I am not sure that how long does it take to let program take note on file
I have hit django for serveral times, but no file at all.

I have added:

DJANGO_LOGGING = {
    "FILE_LOG": False,
    "CONSOLE_LOG": True,
    "MINIFY_CONSOLE_LOG": True,
    "RESPONSE_FIELDS": ('status', 'reason'),
    "DISABLE_EXISTING_LOGGERS":False
}

But I can still see logs/*.log files getting created and being updated with the logdata.

Am I not overriding the variables in the right way?
RESPONSE _FIELDS seems to work fine. Just not the FILE_LOG

@limberger

Logging each request and response in production would make responses slower. Is it possible to disable logging to file while in prod environment? I want to use this plugin for logging in dev/staging environment for debugging purposes.

Hi!
I can make a pull request to create a version with a setting to disable the file logging (to only log to console)? It is ok?

According to your setup.py, django-logging is compatible with Python 3.4 and 3.5. However, it fails on Python 3.4 due to the incorrect assumption that traceback.TracebackException is available—according to the official docs, this object was only added in 3.5. Thus, running django-logging with Python 3.4 results in numerous exceptions that look like this:

Dec 12 16:04:50 hostname gunicorn[20347]: During handling of the above exception, another exception occurred:
Dec 12 16:04:50 hostname gunicorn[20347]: Traceback (most recent call last):
Dec 12 16:04:50 hostname gunicorn[20347]: File "/usr/lib/python3.4/logging/handlers.py", line 71, in emit
Dec 12 16:04:50 hostname gunicorn[20347]: if self.shouldRollover(record):
Dec 12 16:04:50 hostname gunicorn[20347]: File "/usr/lib/python3.4/logging/handlers.py", line 187, in shouldRollover
Dec 12 16:04:50 hostname gunicorn[20347]: msg = "%s\n" % self.format(record)
Dec 12 16:04:50 hostname gunicorn[20347]: File "/home/gcorn/freedompress/lib/python3.4/site-packages/django_logging/handlers.py", line 46, in format
Dec 12 16:04:50 hostname gunicorn[20347]: message = record.msg.to_dict
Dec 12 16:04:50 hostname gunicorn[20347]: File "/home/gcorn/freedompress/lib/python3.4/site-packages/django_logging/log_object.py", line 109, in to_dict
Dec 12 16:04:50 hostname gunicorn[20347]: exception=ErrorLogObject.format_exception(self.exception)
Dec 12 16:04:50 hostname gunicorn[20347]: File "/home/gcorn/freedompress/lib/python3.4/site-packages/django_logging/log_object.py", line 128, in format_exception
Dec 12 16:04:50 hostname gunicorn[20347]: _traceback = traceback.TracebackException.from_exception(exception).exc_traceback
Dec 12 16:04:50 hostname gunicorn[20347]: AttributeError: 'module' object has no attribute 'TracebackException'

Is there any way to configure the JSON output? My company's ELK cluster expects certain fields to have specific names/datatypes in order to be processed by logstash plugins. For example:

 { 
    "timestamp": "2017-01-03T18:46:06Z", # has to be ISO8601
    "loglevel": "debug|info|...",
    "clientip": "192.168.1.1", # for geoip
    "otherfields": {
              .
              .
 }

Hi,

I get a KeyError: 'ContentType' when I return a 204 status code.
Did you encounter problems like this before?
Do you know a fix for this?