Giter VIP home page Giter VIP logo

appenlight-client-python's Introduction

image

appenlight_client

Python Logo

Pyramid Logo

Flask Logo

Django Logo

Installation and Setup

Install the appenlight_client using pip:

pip install appenlight-client

Main Documentation location

The App Enlight developer documentation contains the most up to date information, including implementation guides in popular python web frameworks.

http://getappenlight.com/page/api/main.html

Usage

Before you can use the client inside your application, you first need to navigate to the directory that stores your application configuration and issue the following command:

$ENV/bin/python/appenlight_client makeini appenlight.ini

Usage (example for pyramid or other WSGI pipeline compatible solutions like Zope):

In your INI file, you need to add:

[filter:appenlight_client]
use = egg:appenlight_client
appenlight.config_path = %(here)s/appenlight.ini #optional if you don't want to set APPENLIGHT_INI env var

[pipeline:main]
pipeline =
    .....your other pipeline entries ....
    appenlight_client
    app_name

To minimize configuration complexity, the client will, by default, look for the APPENLIGHT_INI environment variable that will supply the absolute path to the config file.

For a pylons app, you can modify config/middleware.py: import the callable and add these lines:

#exception gathering
# CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)

app = make_appenlight_middleware(app,config)

and add to your ini:

appenlight.config_path = %(here)s/appenlight.ini #optional if you don't want to set APPENLIGHT_INI env var

The App Enlight client provides slow call and datastore timing capabilities; current out-of-the-box libraries supported:

  • most used dbapi2 drivers
  • django templates
  • httplib
  • jinja2 templates
  • mongodb
  • mako templates
  • pysolr
  • requests
  • urllib
  • urllib2
  • urllib3

If for some reason you want to disable the timing of specific library, just set the time value to false.

Configuring appenlight and django

For a django framework, there is separate compatible middleware provided.

Modify your settings file to contain:

import appenlight_client.client as e_client
APPENLIGHT = e_client.get_config()

Additionally, the middleware stack needs to be modified with additional middleware:

MIDDLEWARE_CLASSES = (
    'appenlight_client.django_middleware.AppenlightMiddleware',
    'django.middleware.common.CommonMiddleware',
    ...

Please note that the App Enlight middleware should be placed first in your stack to function properly.

Run your django app providing the APPENLIGHT_INI env variable containing the absolute path to your config file.

Changing default scaffold configuration in Pyramid Web Framework

Default scaffolds in pyramid 1.3 have a section called [app:main] - the App Enlight client expects that you are using wsgi pipeline instead to position itself in it.

The easiest way to accomplish that is to alter your configuration file to look like this:

[app:main] becomes [app:yourappname] 

and inside your configuration, above the [server:main] directive, the following directive should appear:

[pipeline:main]
pipeline =
    ... your other middleware you may have ...
    appenlight_client
    yourappname

Exception views in a Pyramid Web Framework and Appenlight

Pyramid uses exception views to serve nice html templates when an exception occurs. Unfortunately, this means that an exception is handled BEFORE it reaches App Enlight's middleware, so any 500 error data will never get sent to App Enlight.

This is how you can handle error handling inside your error_view:

def error_view(exc, request):
    from appenlight_client.exceptions import get_current_traceback
    from appenlight_client.timing import get_local_storage
    appenlight_storage = get_local_storage()
    stats, slow_calls = appenlight_storage.get_thread_stats()
    traceback = get_current_traceback(skip=1, show_hidden_frames=True, ignore_system_exceptions=True)
    request.environ['appenlight.client'].py_report(request.environ, traceback, message=None,http_status=500, request_stats=stats)
    request.response.status = 500
    return {}

Sensitive data filtering

The client by default blanks out COOKIE,POST, and GET for keys like: 'password','passwd','pwd','auth_tkt'

This behaviour can be altered to filter all kinds of data from the structures that get sent to the server by passing a dotted module name in configuration:

appenlight.filter_callable = foo.bar.baz:callable_name

example:

def callable_name(structure, section=None):
    structure['request']['SOMEVAL'] = '***REMOVED***'
    return structure

App Enlight will try to import foo.bar.baz and use callable_name as the function that accepts parameters (structure, section) and returns altered data structure.

Please note that this functionality can be used to alter things like the App Enlight grouping mechanism; you can set this variable based on values present in the structure generated by the client.

appenlight_client is BSD licensed, consult LICENSE for details.

client source: https://github.com/AppEnlight/appenlight-client-python

appenlight-client-python's People

Contributors

abnerpc avatar aop avatar djeebus avatar ergo avatar georgski avatar haloween avatar lisaq avatar luk156 avatar malthe avatar slav0nic avatar virhilo avatar volksman avatar wichert avatar ztane avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

appenlight-client-python's Issues

Allow Blacklisting of environ_keys

Currently only whitelisting environ variables is allowed. However, I cannot blacklist, for instance, HTTP_COOKIE without modifying the library locally. This is important for me (and perhaps others) because I am not okay with allowing signed tokens to be seen by staff reviewing these logs. I can make a PR soon probably for this, but this would be extremely helpful.

Using a single view decorator does not work

If you do something like this:

@view_config(route_name='xyz', decorator=some_func)
def my_view(request):
    pass

The appenlight client breaks:

  File "/Users/wichert/Library/buildout/eggs/pyramid-1.5-py2.7.egg/pyramid/config/__init__.py", line 929, in scan
    ignore=ignore)
  File "/Users/wichert/Library/buildout/eggs/venusian-1.0a8-py2.7.egg/venusian/__init__.py", line 208, in scan
    invoke(modname, name, ob)
  File "/Users/wichert/Library/buildout/eggs/venusian-1.0a8-py2.7.egg/venusian/__init__.py", line 167, in invoke
    callback(self, name, ob)
  File "/Users/wichert/Library/buildout/eggs/pyramid-1.5-py2.7.egg/pyramid/view.py", line 190, in callback
    config.add_view(view=ob, **settings)
  File "/Users/wichert/Library/buildout/eggs/appenlight_client-0.6.7-py2.7.egg/appenlight_client/ext/pyramid_tween.py", line 100, in wrapper
    kwargs['decorator'].append(wrap_pyramid_view_name)
AttributeError: 'limit_to_team_types' object has no attribute 'append'

Using a tuple as a decorator, as used in the Pyramid documentation, also fails because a tuple does not have an append method.

Rewriting wrap_view_config like this should work:

def wrap_view_config(appenlight_callable):
    @wraps(appenlight_callable)
    def wrapper(*args, **kwargs):
        if kwargs.get('decorator') is None:
            if can_append_decorator:
                kwargs['decorator'] = wrap_pyramid_view_name
        else:
            if can_append_decorator:
                current = kwargs['decorator']
                if isinstance(current, (list, tuple)):
                    kwargs['decorator'] = list(current) + [wrap_pyramid_view_name]
                else:
                    kwargs['decorator'] = (current, wrap_pyramid_view_name)
        return appenlight_callable(*args, **kwargs)

    return wrapper

Invalid status code in report

Django - page returns 404 but AppEnlight shows HTTP status as 500. See attached image
appenlight

I've also tested page with firebug and code returned is 404

AttributeError in pyramid_tween

appenlight don't work correct with bound method views

Traceback (most recent call last):
File "/mnt/data/work/zojax/quickpyramid/lib/python3.7/site-packages/websauna/system/auth/tweens.py", line 60, in __call__
    response = self.handler(request)
File "quick/dashboard/tweens.py", line 54, in __call__
    response = self.handler(request)
File "pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
File "pyramid/view.py", line 657, in _call_view
    response = view_callable(context, request)
File "pyramid/viewderivers.py", line 384, in authdebug_view
    return view(context, request)
File "pyramid/viewderivers.py", line 514, in csrf_view
    return view(context, request)
File "appenlight_client/ext/pyramid_tween.py", line 99, in view_callable_wrapper
    original_view._appenlight_name = view_name
AttributeError: 'method' object has no attribute '_appenlight_name'

original_view is
<bound method Sitemap.render of <quick.core.sitemap.Sitemap object at 0x7f48755c51d0>>

Pyramid support bounded method, but you can't set attr on it, internally i find some workaround for solve this: https://github.com/Pylons/pyramid/blob/587fe72fae0efda3a860d37a1ea2449a41dab622/src/pyramid/viewderivers.py#L94

Sample View for reproduce error:

sm = Sitemap()
...
config.add_route("sitemap", "/sitemap.xml")
config.add_view(sm.render, route_name="sitemap", renderer="core/sitemap.xml")

Log entries should use appenlight.tags

Currently appenlight has two completely separate ways to define tags:

  • the extrakeyword parameter for logging
  • appenlight.tags in the environ for everything else

This means that to get the same tags in logging you have to use extra=<something> to every single log statement in your code. That quickly becomes very painful to do and maintain.

I would like to see logging use appenlight.tags as well, allowing you to use the extra parameter to add or override tags.

'template' key in jinja2 renderer breaks the client

We run into an issue where having a 'template' key in the jinja2 context data dictionary breaks when using appenlight in pyramid.

Looks like appenlight decorator tries to do an updade on the dictionary with that key duplicated on such case.

I renamed the data key to something else and it worked.

Locals does not Include Function Arguments

The appenlight locals display does not seem to include values passed in as function arguments.

What additional information should be provided to diagnose this problem?

ignore_slow_paths and ignore_paths settings must be list

This ini settings must be list, aslist() missed

self.config['ignore_slow_paths'] = \
config.get('appenlight.ignore_slow_paths', [])
self.config['ignore_paths'] = \
config.get('appenlight.ignore_paths', [])

Also appenlight.cookie_keys_whitelist have strange code where is reused itself as default value

cookie_whitelist = aslist(
config.get('appenlight.cookie_keys_whitelist',
config.get('appenlight.cookie_keys_whitelist')), ',')

setup.py should not import anything

If you try and install into a fresh virtualenv it will fail because the pass of setup.py required to find the requirements imports appenlight_client which imports the actual client and generates an ImportError.

Collecting appenlight-client==0.6.18 (from -r /app/requirements.txt (line 26))
  Downloading appenlight_client-0.6.18.tar.gz (46kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-_jxiewns/appenlight-client/setup.py", line 1, in <module>
        import appenlight_client
      File "/tmp/pip-build-_jxiewns/appenlight-client/appenlight_client/__init__.py", line 18, in <module>
        from appenlight_client.client import make_appenlight_middleware
      File "/tmp/pip-build-_jxiewns/appenlight-client/appenlight_client/client.py", line 42, in <module>
        from webob import Request
    ImportError: No module named 'webob'

500 errors when I set APPENLIGHT_KEY

On Django Packages, when I set the APPENLIGHT_KEY I get 500 errors. This is a sample traceback:

File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 188, in get_response
  response = middleware_method(request, response)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.13.1.31/newrelic/api/object_wrapper.py", line 237, in __call__
  self._nr_instance, args, kwargs, **self._nr_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-1.13.1.31/newrelic/hooks/framework_django.py", line 304, in wrapper
  return wrapped(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/appenlight_client/django_middleware.py", line 110, in process_response
  slow_calls=slow_calls)
File "/app/.heroku/python/lib/python2.7/site-packages/appenlight_client/client.py", line 326, in py_report
  include_params=True)
File "/app/.heroku/python/lib/python2.7/site-packages/appenlight_client/client.py", line 530, in create_report_structure
  http_status)
File "/app/.heroku/python/lib/python2.7/site-packages/appenlight_client/client.py", line 449, in process_environ
  if key.startswith('appenlight.') \
AttributeError: ‘tuple’ object has no attribute ‘startswith’

<WSGIRequest path:/, GET:<QueryDict: {}>, POST:<QueryDict: {}>, COOKIES:{'csrftoken': ‘Kmz3Ne2yGgwYQWdUPt6yMGPvgvBX1qET’,

'sessionid': 'cc6e3535dc26cf3c53a0122b235b3263'},
META:{'CSRF_COOKIE': ‘Kmz3Ne2yGgwYQWdUPt6yMGPvgvBX1qET’,

'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8',
'HTTP_CACHE_CONTROL': 'max-age=0',
'HTTP_CONNECTION': 'close',
'HTTP_COOKIE': 'csrftoken=Kmz3Ne2yGgwYQWdUPt6yMGPvgvBX1qET; sessionid=cc6e3535dc26cf3c53a0122b235b3263',
'HTTP_HOST': 'www.djangopackages.com',
'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36',
'HTTP_X_FORWARDED_FOR': '64.134.45.71',
'HTTP_X_FORWARDED_PORT': '443',
'HTTP_X_FORWARDED_PROTO': 'https',
'HTTP_X_REQUEST_ID': '6be66f67-4653-4fa2-ac66-e093854c02e3',
'HTTP_X_REQUEST_START': '1392654392576',
'PATH_INFO': u'/',
'QUERY_STRING': '',
'RAW_URI': '/',
'REMOTE_ADDR': '64.134.45.71',
'REMOTE_PORT': '80',
'REQUEST_METHOD': 'GET',
'SCRIPT_NAME': u'',
'SERVER_NAME': 'www.djangopackages.com',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SOFTWARE': 'gunicorn/0.14.2',
'appenlight.client': <appenlight_client.client.Client object at 0x35612d0>,
'appenlight.post_vars': <QueryDict: {}>,
'appenlight.request_id': '7a44de6a-e9c2-4c2c-bdfe-2121c56c4fa4',
'appenlight.username': u'2',
'appenlight.view_name': 'homepage/views:homepage.GET',
'gunicorn.socket': <socket._socketobject object at 0x313bec0>,
'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f12d18fe1e0>,
'wsgi.file_wrapper': <class gunicorn.http.wsgi.FileWrapper at 0x3128db8>,
'wsgi.input': <newrelic.api.web_transaction.WSGIInputWrapper object at 0x3547c50>,
'wsgi.multiprocess': True,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0),
(u'reversion.revision_middleware_active', <reversion.middleware.RevisionMiddleware object at 0x35517d0>): True}>

client lost its API key

I just saw this in the logs of a docker container:

02:57:02 INFO  [appenlight_client.transports.requests] sending out 2 entries to /api/metrics
02:57:02 INFO  [requests.packages.urllib3.connectionpool] Starting new HTTPS connection (1): api.appenlight.com
03:22:37 INFO  [appenlight_client.transports.requests] sending out 1 entries to /api/metrics
03:22:37 INFO  [requests.packages.urllib3.connectionpool] Starting new HTTPS connection (1): api.appenlight.com
06:33:27 INFO  [appenlight_client.transports.requests] sending out 1 entries to /api/metrics
06:33:27 INFO  [requests.packages.urllib3.connectionpool] Starting new HTTPS connection (1): api.appenlight.com
06:36:24 WARNI [curvetips.site.app] No AppEnlight API key or config file configured.
06:36:24 WARNI [curvetips.site.app] No AppEnlight API key or config file configured.
06:39:04 WARNI [curvetips.site.app] No AppEnlight API key or config file configured.
06:39:04 WARNI [curvetips.site.app] No AppEnlight API key or config file configured.

This suggests that the client forgot about its API key.

celery shell broken by appenlight

IPython.utils.traitlets.TraitError: The 'db' trait of a HistoryManager instance must be a Connection or None, but a value of class 'appenlight_client.timing.timing_dbapi2.TimerWrapper' (i.e. <appenlight_client.timing.timing_dbapi2.TimerWrapper object at 0x7eee350>) was specified.

Appenlight replace the ipython db with a timing proxy, however the timing proxy doesn't work with iPython.

Investigate if some versions of pg8000 fail to get decorated properly

  File "/home/affiliate/python/lib/python2.7/site-packages/appenlight_client/client.py", line 625, in make_appenlight_middleware
    client = Client(config)
  File "/home/affiliate/python/lib/python2.7/site-packages/appenlight_client/client.py", line 197, in __init__
    appenlight_client.timing.register_timing(self.config)
  File "/home/affiliate/python/lib/python2.7/site-packages/appenlight_client/timing/__init__.py", line 162, in register_timing
    dbapi2.add_timing(mod, min_time)
  File "/home/affiliate/python/lib/python2.7/site-packages/appenlight_client/timing/timing_dbapi2.py", line 178, in add_timing
    if hasattr(module.DBAPI.connect, '_e_attached_wrapper'):
AttributeError: 'module' object has no attribute 'DBAPI'

django

I added middleware 'django.middleware.security.SecurityMiddleware', I also have my custom middleware. When execute request.user.is_authenticated() in custom middleware exeption appears AttributeError __exit__

Keeping all service timing/logs for slow requests?

Example -- a badly behaved Django request ends up making 200 SQL queries and takes 2 seconds (so triggers slow-request handling):

  • you can see that 200 queries happened in the breakdown bar on the left
  • but each query is individually quick (eg. taking 10ms)
  • so there's no breakdown on what it's actually doing in your slow request
  • so it's harder than it needs to be to diagnose

I guess the approach is to accumulate all log/timing info before deciding whether to report the slow request?

Setting appenlight.timing.{library} = 0 for everything appears to achieve that without triggering slow-reporting for individual service calls that exceed the timing threshold. (well, 0.001 -- 0 doesn't actually work in practice)

For logging, logging_on_error doesn't appear to be used at all? -- either the logging is always on or off.

Am I missing something?

UnboundLocalError: local variable 'e' referenced before assignment

At the line mentioned below, if no transport is set in configuration, an UnboundLocalError is raised:

  File "/home/anchor/env/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "build/bdist.linux-x86_64/egg/appenlight_client/client.py", line 616, in make_appenlight_middleware
  File "build/bdist.linux-x86_64/egg/appenlight_client/client.py", line 635, in make_appenlight_middleware_with_client
  File "build/bdist.linux-x86_64/egg/appenlight_client/client.py", line 85, in __init__
  File "build/bdist.linux-x86_64/egg/appenlight_client/client.py", line 224, in reinitialize
UnboundLocalError: local variable 'e' referenced before assignment

This was happening in one of our projects and the solution was to add a transport to the config file to preventing it from happening:

appenlight.transport = appenlight_client.transports.requests:HTTPTransport

Perhaps somebody forgot to remove the e variable from the logging in an old try/except block.

self.config['transport'], e)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.