Giter VIP home page Giter VIP logo

iopipe-python's Introduction

IOpipe Agent for JavaScript


Coverage Status npm version styled with prettier semantic-release

IOpipe is a serverless DevOps platform for organizations building event-driven architectures in AWS Lambda. IOpipe captures crucial, high-fidelity metrics for each Lambda function invocation. This data powers a flexibile and robust development and operations experience with features including tracing, profiling, custom metrics, and low-latency alerts. Get started today to quickly and confidently gain superior observability, identify issues, and discover anomalies in your connected applications.

Note: this library is a lower-level implementation than the package you might likely be looking for. Enjoy pre-bundled plugins like tracing and event info with @iopipe/iopipe

Installation

Install using your package manager of choice,

npm install @iopipe/core

or

yarn add @iopipe/core

If you are using the Serverless Framework to deploy your lambdas, check out our serverless plugin.

Usage

Configure the library with your project token (register for access), and it will automatically monitor and collect metrics from your applications running on AWS Lambda.

Example:

const iopipeLib = require('@iopipe/core');

const iopipe = iopipeLib({ token: 'PROJECT_TOKEN' });

exports.handler = iopipe((event, context) => {
  context.succeed('This is my serverless function!');
});

Custom metrics

You may add custom metrics to an invocation using context.iopipe.metric to add either string or numerical values. Keys have a maximum length of 256 characters, and string values are limited to 1024.

Example:

const iopipeLib = require('@iopipe/core');

const iopipe = iopipeLib({ token: 'PROJECT_TOKEN' });

exports.handler = iopipe((event, context) => {
  context.iopipe.metric('key', 'some-value');
  context.iopipe.metric('another-key', 42);
  context.succeed('This is my serverless function!');
});

Labels

You can label invocations using context.iopipe.label to label an invocation with a string value, with a limit of 128 characters.

Example:

const iopipeLib = require('@iopipe/core');

const iopipe = iopipeLib({ token: 'PROJECT_TOKEN' });

exports.handler = iopipe((event, context) => {
  context.iopipe.label('something-important-happened');
  context.succeed('This is my serverless function!');
});

Configuration

Methods

You can configure your iopipe setup through one or more different methods - that can be mixed, providing a config chain. The current methods are listed below, in order of precendence. The module instantiation object overrides all other config values (if values are provided).

  1. Module instantiation object
  2. IOPIPE_* environment variables
  3. An .iopiperc file
  4. An iopipe package.json entry
  5. An extends key referencing a config package
  6. Default values

Options

token (string: required)

If not supplied, the environment variable $IOPIPE_TOKEN will be used if present. Find your project token.

debug (bool: optional = false)

Debug mode will log all data sent to IOpipe servers to STDOUT. This is also a good way to evaluate the sort of data that IOpipe is receiving from your application. If not supplied, the environment variable $IOPIPE_DEBUG will be used if present.

const iopipe = require('@iopipe/core')({
  token: 'PROJECT_TOKEN',
  debug: true
});

exports.handler = iopipe((event, context, callback) => {
  // Do things here. We'll log info to STDOUT.
});

networkTimeout (int: optional = 5000)

The number of milliseconds IOpipe will wait while sending a report before timing out. If not supplied, the environment variable $IOPIPE_NETWORK_TIMEOUT will be used if present.

const iopipe = require('@iopipe/core')({ token: 'PROJECT_TOKEN', networkTimeout: 30000})

timeoutWindow (int: optional = 150)

By default, IOpipe will capture timeouts by exiting your function 150ms early from the AWS configured timeout, to allow time for reporting. You can disable this feature by setting timeoutWindow to 0 in your configuration. If not supplied, the environment variable $IOPIPE_TIMEOUT_WINDOW will be used if present.

const iopipe = require('@iopipe/core')({ token: 'PROJECT_TOKEN', timeoutWindow: 0})

plugins (array: optional)

Note that if you use the @iopipe/iopipe package, you get our recommended plugin set-up right away. Plugins can extend the functionality of IOpipe in ways that best work for you. Follow the guides for the plugins listed below for proper installation and usage on the @iopipe/core library:

Example:

const tracePlugin = require('@iopipe/trace');

const iopipe = require('@iopipe/core')({
  token: 'PROJECT_TOKEN',
  plugins: [tracePlugin()]
});

exports.handler = iopipe((event, context, callback) => {
  // Run your fn here
});

enabled (boolean: optional = True)

Conditionally enable/disable the agent. The environment variable $IOPIPE_ENABLED will also be checked.

url (string: optional)

Sets an alternative URL to use for the IOpipe collector. The environment variable $IOPIPE_COLLECTOR_URL will be used if present.

RC File Configuration

Not recommended for webpack/bundlers due to dynamic require.

You can configure iopipe via an .iopiperc RC file. An example of that is here. Config options are the same as the module instantiation object, except for plugins. Plugins should be an array containing mixed-type values. A plugin value can be a:

  • String that is the name of the plugin
  • Or an array with plugin name first, and plugin options second
{
  "token": "wow_token",
  "plugins": [
    "@iopipe/trace",
    ["@iopipe/profiler", {"enabled": true}]
  ]
}

IMPORTANT: You must install the plugins as dependencies for them to load properly in your environment.

package.json Configuration

Not recommended for webpack/bundlers due to dynamic require.

You can configure iopipe within a iopipe package.json entry. An example of that is here. Config options are the same as the module instantiation object, except for plugins. Plugins should be an array containing mixed-type values. A plugin value can be a:

  • String that is the name of the plugin
  • Or an array with plugin name first, and plugin options second
{
  "name": "my-great-package",
  "dependencies": {
    "@iopipe/trace": "^0.2.0",
    "@iopipe/profiler": "^0.1.0"
  },
  "iopipe": {
    "token": "wow_token",
    "plugins": [
      "@iopipe/trace",
      ["@iopipe/profiler", {"enabled": true}]
    ]
  }
}

IMPORTANT: You must install the plugins as dependencies for them to load properly in your environment.

Extends Configuration

Not recommended for webpack/bundlers due to dynamic require.

You can configure iopipe within a package.json or rc file by referencing a extends config package. An example of that is here. Config options are the same as the module instantiation object, except for plugins. Plugins should be an array containing mixed-type values. A plugin value can be a:

  • String that is the name of the plugin
  • Or an array with plugin name first, and plugin options second

For an example of a config package, check out @iopipe/config.

IMPORTANT: You must install the config package and plugins as dependencies for them to load properly in your environment.

License

Apache 2.0

iopipe-python's People

Contributors

alexdebrie avatar ewindisch avatar katiebayes avatar kolanos avatar marknca avatar mislavcimpersak avatar nodebotanist avatar pselle avatar vemel avatar wohlgejm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

iopipe-python's Issues

Add timestampEnd field

Add timestampEnd to reporting for process tracking, don't forget to remove from optional_fields in the schema test.

Auto-measure creating end:iopipe values

This occurred both when using no marks and adding marks

example of data seen:

   {
            "duration": 0,
            "entryType": "mark",
            "name": "end:iopipe",
            "startTime": 149761.21127399988,
            "timestamp": 1522297584199
        }

Reproduction:

  • Use tracing plugin with auto_measure set to true

Add acceptance tests for event info plugin

Description

Steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Update documentation for arguments

Positionally, the arguments are documented as:

  • token
  • debug
  • url

in the code, the order of debug and url are in fact the inverse. Update documentation for arguments, consider adding tests for using kwargs

Acceptance tests on master

We have acceptance tests on the node agent that run on the master branch to catch breakages. Let's add those here as well.

Disable agent if not running on Linux

If the agent is running on OSX or some other OS it should disable itself to avoid raising errors while attempting to collect Linux-specific telemetry data.

Dependencies not specified

Unless I'm doing something wrong the pip dependencies don't seem to specified anywhere so it's not possible to simply run pip install iopipe and start using the library.

I can reproduce this like so (with pipenv):

  • Create a fresh virtual env pipenv --three
  • Install iopipe pip install iopipe
  • Run a test script that imports iopipe python -c 'import iopipe'

I get the following error as I don't have the six library installed:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/james/.local/share/virtualenvs/test-gXZ6kH7S/lib/python3.6/site-packages/iopipe/__init__.py", line 1, in <module>
    from .iopipe import IOpipe  # noqa
  File "/Users/james/.local/share/virtualenvs/test-gXZ6kH7S/lib/python3.6/site-packages/iopipe/iopipe.py", line 8, in <module>
    from .context import ContextWrapper
  File "/Users/james/.local/share/virtualenvs/test-gXZ6kH7S/lib/python3.6/site-packages/iopipe/context.py", line 4, in <module>
    from six import string_types
ModuleNotFoundError: No module named 'six'

I believe the six dependency and any others should be included in setup.py using install_requires?

use botocore.vendored.requests instead of implying an app depedency

given the execution environment, adding an additional dependency seems extraneous, given the platform already has the underlying library just by switching the import location. perhaps not meaningful given the common installation of this library, but worth a consideration.

Configurable network timeout

The agent currently accepts an network_timeout option but does not use it. This timeout should be passed to requests.

Auto-measure should default to true

auto_measure is documented as if it needs to be set to true in order to auto measure -- this is inconsistent with the node plugin, and we should default to true, letting users override that and add measure calls as an enhancement, not the first recommendation.

Run system reads in parallel when possible

The system reads can be run in the background via multiprocessing.Queue and collected at the end of the invocation. This way the filesystem reads are non-blocking for the handler itself. This also serves as an optimization for system reads in general.

Mock system data

Fork of #97, after shipping the option to disable, we should still allow invokes to run locally/report to IOpipe if they want using mock data. More details available on #97

Python agent not working for local invoke

I am not sure if Python is working for local invokes.

$ python main.py 
Error reporting metrics to IOpipe. sys.meta_path must be a list of import hooks

using main.py from qa-events.

Problems installing with pip

It looks like the requirements.txt is missing from the package uploaded to pip

Collecting iopipe (from -r requirements.txt (line 1))
  Using cached iopipe-0.1.2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/download.py", line 421, in get_file_content
        with open(url, 'rb') as f:
    FileNotFoundError: [Errno 2] No such file or directory: './requirements.txt'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/3t/vw8l_45s3bsb4rz3c2l9wvh00000gn/T/pip-build-3822khjv/iopipe/setup.py", line 8, in <module>
        reqs = [str(ir.req) for ir in install_reqs]
      File "/private/var/folders/3t/vw8l_45s3bsb4rz3c2l9wvh00000gn/T/pip-build-3822khjv/iopipe/setup.py", line 8, in <listcomp>
        reqs = [str(ir.req) for ir in install_reqs]
      File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_file.py", line 84, in parse_requirements
        filename, comes_from=comes_from, session=session
      File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/download.py", line 425, in get_file_content
        'Could not open requirements file: %s' % str(exc)
    pip.exceptions.InstallationError: Could not open requirements file: [Errno 2] No such file or directory: './requirements.txt'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/3t/vw8l_45s3bsb4rz3c2l9wvh00000gn/T/pip-build-3822khjv/iopipe/```

Logging Plugin

A plugin to attach a logging handler to upload logs to IOpipe.

Context wrapper

The agent should wrap contexts similar to the way the Node.js agent does it to enable features such as trace markers.

Memory reported does not match what's in Cloudwatch

Description

The memory usage displayed in IOpipe shows as much lower than the report of MaxMemory in Cloudwatch. One example from a user shows IOpipe reporting 45.1 mb / 384 mb for an invocation. The same invocation in Cloudwatch reports Memory Size: 384 MB Max Memory Used: 272 MB

Steps to reproduce the issue:
1.Run an invocation and compare between IOpipe and Cloudwatch

Support writing logs to /tmp

The Logger Plugin should optionally support logging to the lambda function's /tmp directory. For functions that are memory intensive, this will likely be preferable to an in-memory buffer.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.