Giter VIP home page Giter VIP logo

lambda-uploader's Introduction

lambda-uploader

Provides a quick command line utility for packaging and publishing Python AWS Lambda functions. This is a work in progress and pull requests are always welcome.

Installation

The latest release of lambda-uploader can be installed via pip:

pip install lambda-uploader

An alternative install method would be manually installing it leveraging setup.py:

git clone https://github.com/rackerlabs/lambda-uploader
cd lambda-uploader
python setup.py install

Configuration File

The lambda uploader expects a directory with, at a minimum, your lambda function and a lambda.json file. It is not necessary to set requirements in your config file since the lambda uploader will also check for and use a requirements.txt file.

Please note that you can leave the vpc object out of your config if you want your lambda function to use your default VPC and subnets. If you wish to use your lambda function inside a specific VPC, make sure you set up the role correctly to allow this.

Note also the ignore entry is an array of regular expression strings used to match against the relative paths - be careful to quote accordingly. For example, a traditional *.txt "glob" is matched by the JSON string: ".*\\.txt$" (or just "\\.txt$").

Example lambda.json file:

{
  "name": "myFunction",
  "description": "It does things",
  "region": "us-east-1",
  "runtime": "python2.7",
  "handler": "function.lambda_handler",
  "role": "arn:aws:iam::00000000000:role/lambda_basic_execution",
  "requirements": ["pygithub"],
  "ignore": [
    "circle\\.yml$",
    "\\.git$",
    "/.*\\.pyc$"
  ],
  "timeout": 30,
  "memory": 512,
  "vpc": {
    "subnets": [
      "subnet-00000000"
    ],
    "security_groups": [
      "sg-00000000"
    ]
  },
  "tracing": {
    "Mode": "Active"
  }
}

You can also optionally setup a subscription to a Kinesis stream for your lambda using the subscription field as in the following sample configuration.

{
  "name": "myFunction",
  "description": "It does things",
  "region": "us-east-1",
  "runtime": "python2.7",
  "handler": "function.lambda_handler",
  "role": "arn:aws:iam::00000000000:role/lambda_basic_execution",
  "requirements": ["pygithub"],
  "timeout": 30,
  "memory": 512,
  "subscription": {
    "kinesis": {
      "stream": "arn:aws:kinesis:eu-west-1:000000000000:stream/services",
      "batch_size": 10
    }
  }
}

Command Line Usage

To package and upload simply run the command from within your lambda directory or with the directory as an option.

lambda-uploader ./myfunc

To specify an alternative profile that has been defined in ~/.aws/credentials use the --profile parameter.

lambda-uploader --profile=alternative-profile

To specify an alternative, prexisting virtualenv use the --virtualenv parameter.

lambda-uploader --virtualenv=~/.virtualenv/my_custom_virtualenv

To omit using a virtualenv use the --no-virtualenv parameter.

lambda-uploader --no-virtualenv

To inject any other additional files, use the --extra-file EXTRA_FILE parameter.

lambda-uploader --extra-file ~/stuff_for_lambda_packages

If you would prefer to upload another way you can tell the uploader to ignore the upload. This will create a package and leave it in the project directory.

lambda-uploader --no-upload ./myfunc

To publish a version without an alias you would pass the the publish flag.

lambda-uploader -p ./myfunc

If you would like to alias your upload you can pass the alias with the alias flag. The function description will be used when an alias-description is not provided.

lambda-uploader --alias myAlias --alias-description 'My alias description' ./myfunc

If you would prefer to build the package manually and just upload it using uploader you can ignore the build. This will upload lambda_function.zip file.

lambda-uploader --no-build

lambda-uploader's People

Contributors

abetomo avatar agdude avatar azatoth avatar binarymatt avatar brandond avatar calebgroom avatar declension avatar deviavir avatar dude051 avatar gdelvalle avatar havenwang avatar jarosser06 avatar martinb3 avatar moomeen avatar nicholasjstock avatar not-raspberry avatar odeckmyn avatar ps-jay avatar quiver avatar rriifftt avatar ryandub avatar stavxyz avatar tomislater avatar zkanda avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lambda-uploader's Issues

Extra copy of virtualenv is included in every zip file

It appears that we're storing an entire extra copy of the virtualenv in .lambda_package/venv, meaning that .lambda_package/venv/lib/python2.7/site-packages/ is a duplicate of .lambda_package/lambda_package. I think we can skip the entire venv directory from being included in the .zip.

Easy sharing of code between lambda functions

Is there a way to share python code between lambdas, without having to create an additional package to install via requirements?

e.g., in the Serverless framework, there's the concept of a "lib" folder which can contain modules shared between multiple lambdas.

Option zip and upload only the folder contents

Right now, uploading grabs a bunch of stuff I don't need (git, PyGitHub, etc). This creates a fairly large zip file, hence, larger lambda storage.

Not being strong with Python, I'm guessing the extra stuff relates to using virtualenv. I don't need all those dependencies, and would prefer to pip install to the local directory. Could this be added as a new switch? Or is there something I'm missing and there's some other way for me to skip them?

Python 2.6 incompatible - I think that lambda-uploader complain

Hi there,

I've just tried lambda-uploader on CentOS 6... with Python 2.6.. ooops!
It of course doesn't work very well - but understandably - Python 2.6 is very out-of-date and Lambda itself uses 2.7.

I think that lambda-uploader should detect a non-Python 2.7 environment and refuse to work.
Else, it seems to work, but ends up producing a zero-byte lambda_function.zip - which is just confusing.

If you think this is a good idea, I'm happy to work on a PR to implement.

Thoughts?

Thanks for the Great Work!

I just wanted to leave a note to thank you for this project. It is great and inspired me to use it to solve some of our Lambda issues by wrapping it in a Flask like framework called Lambada to simplify managing multiple lambdas in one project.

Failure when installing requirements from lambda.json

Nice tool, 1st off.

I'm getting errors when I specify a list of requirements in lambda.json.

Here's my redacted config:

{
  "name": "myFunction",
  "description": "Does stuff",
  "region": "us-east-1",
  "handler": "my_function.lambda_handler",
  "timeout": 30,
  "memory": 1024,
  "requirements": ["arrow==0.7.0", "pyhorn==0.5.0", "pyloggly==0.1.0"]
}

Here's the output:

λ Building Package
DEBUG:lambda_uploader.package:Virtualenv stdout: Using real prefix '/home/jluker/.pyenv/versions/2.7.10'
New python executable in /home/jluker/projects/dce-elk/.lambda_package/venv/bin/python2.7
Also creating executable in /home/jluker/projects/dce-elk/.lambda_package/venv/bin/python
Installing setuptools, pip, wheel...done.

DEBUG:lambda_uploader.package:Virtualenv stderr: 
DEBUG:lambda_uploader.package:Installing requirements found [u'arrow==0.7.0', u'pyhorn==0.5.0', u'pyloggly==0.1.0'] in config
DEBUG:lambda_uploader.package:Pip cmd: ['/home/jluker/projects/dce-elk/.lambda_package/venv/bin/pip', 'install', u'arrow==0.7.0 pyhorn==0.5.0 pyloggly==0.1.0']
DEBUG:lambda_uploader.package:Pip stdout: 
DEBUG:lambda_uploader.package:Pip stderr: Exception:
Traceback (most recent call last):
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/commands/install.py", line 282, in run
    wheel_cache
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/basecommand.py", line 272, in populate_requirement_set
    wheel_cache=wheel_cache
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/req/req_install.py", line 213, in from_line
    wheel_cache=wheel_cache, constraint=constraint)
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/req/req_install.py", line 67, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2980, in parse
    reqs = list(parse_requirements(s))
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2924, in parse_requirements
    "version spec")
  File "/home/jluker/projects/dce-elk/.lambda_package/venv/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2900, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])
RequirementParseError: Expected ',' or end-of-list in arrow==0.7.0 pyhorn==0.5.0 pyloggly==0.1.0 at  pyhorn==0.5.0 pyloggly==0.1.0

⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "/home/jluker/envs/dce-elk/lib/python2.7/site-packages/lambda_uploader/shell.py", line 124, in main
    _execute(args)
  File "/home/jluker/envs/dce-elk/lib/python2.7/site-packages/lambda_uploader/shell.py", line 53, in _execute
    pkg = package.build_package(pth, cfg.requirements)
  File "/home/jluker/envs/dce-elk/lib/python2.7/site-packages/lambda_uploader/package.py", line 35, in build_package
    pkg.install_requirements(requirements)
  File "/home/jluker/envs/dce-elk/lib/python2.7/site-packages/lambda_uploader/package.py", line 94, in install_requirements
    raise Exception('pip returned unsuccessfully')
Exception: pip returned unsuccessfully

Fatal error: local() encountered an error (return code 1) while executing 'lambda-uploader --no-upload -VV --profile test'

Aborting.

I added an extra LOG.debug in there to see the actual pip subprocess cmd. It's puzzling because if I execute same command in the shell it works fine.

Support for VPC Configs

Now that you can specify VPC resources this should be something that is included in the configuration and possibly command line as well.

getting config exception in lambda..

Hi there,

I am getting below given exception on triggering lambda which is uploaded thru lambda-uploader.

The specified config file (/home/sbx_user1051/.aws/config) could not be found.: ConfigNotFound
Traceback (most recent call last):
s3 = boto3.client('s3')
File "/var/task/boto3/init.py", line 79, in client
return _get_default_session().client(_args, *_kwargs)
File "/var/task/boto3/init.py", line 69, in _get_default_session
setup_default_session()
File "/var/task/boto3/init.py", line 33, in setup_default_session
DEFAULT_SESSION = Session(**kwargs)
File "/var/task/boto3/boto3/session.py", line 80, in init
self._setup_loader()
File "/var/task/boto3/boto3/session.py", line 112, in _setup_loader
self._loader = self._session.get_component('data_loader')
File "/var/task/botocore/session.py", line 691, in get_component
return self._components.get_component(name)
File "/var/task/botocore/session.py", line 885, in get_component
self._components[name] = factory()
File "/var/task/botocore/session.py", line 179, in
lambda: create_loader(self.get_config_variable('data_path')))
File "/var/task/botocore/session.py", line 259, in get_config_variable
elif self._found_in_config_file(methods, var_config):
File "/var/task/botocore/session.py", line 280, in _found_in_config_file
return var_config[0] in self.get_scoped_config()
File "/var/task/botocore/session.py", line 342, in get_scoped_config
profile_map = self._build_profile_map()
File "/var/task/botocore/session.py", line 214, in _build_profile_map
self._profile_map = self.full_config['profiles']
File "/var/task/botocore/session.py", line 369, in full_config
self._config = botocore.configloader.load_config(config_file)
File "/var/task/botocore/botocore/configloader.py", line 105, in load_config
parsed = raw_config_parse(config_filename)
File "/var/task/botocore/botocore/configloader.py", line 126, in raw_config_parse
raise botocore.exceptions.ConfigNotFound(path=path)
ConfigNotFound: The specified config file (/home/sbx_user1051/.aws/config) could not be found.

Ignore entries are regexes, not globs

In the example config and elsewhere the ignore section is treated like a list of filesystem globs (e.g. *.txt), whereas the code suggests these are in fact regular expressions matching on the path.

The examples all work as intended (largely because r'.' matches the character .) but would also match unwanted (albeit strange) files, e.g. Xgit or /Xpyc.

Elsewhere in the code, arbitrary filenames should probably be escaped when added to this list, in case they contain special characters (e.g. . but also []()-*? etc) and start matching incorrectly.

I've got (most of) a patch for this, if that helps.

Non-python lambda upload

What do you think about using the tool to upload non-python lambdas?
I'd like to upload java one. The file is being created with maven, so in the end it would be uploading arbitrary zip - so the opposite of --no-upload.
What about --only-upload option?

Ability to create a role with inline policy at upload time

I'm wanting to use lambda-uploader for work stuff instead of my personal project boxidau/alpha, however I can't define a lambda function role policy in the config file with lambda-uploader.

Here's my implementation of this
https://github.com/boxidau/alpha/blob/master/alpha.py#L105-L137

and example config:
https://github.com/boxidau/alpha/blob/master/example_project/test-lambda2/lambda.json#L8-L33

Any chance this functionality could make it into lambda-uploader?

General use

I'm investigating using this tool instead of my bash script.

Question 1

My directory structure is:

OVSensorReadingsProcessor/
  requirements.txt
  handler.py
  ov_sensor_readings_processor/
     utils.py
     processor.py
     ..etc..
  venv/
     .../site-packages/

My scripts pulls in handler.py, my 'ov_sensor_readings_processor' module files, and site-packages. So, first, I called: lambda-uploader ./ That put everything in my project into the .zip, so I copied *.py to a dist directory like:

OVSensorReadingsProcessor/
  dist/
    handler.py
    ov_sensor_readings_processor/
      utils.py
      processor.py
      ..etc..

and then called: lambda-uploader.py dist/ which included only what I wanted in the zip file. Is this the intended directory structure?

Question 2

Still missing from the zip file was my ov_sensor_readings_processor directory. Does the script not recurse down and include everything under dist? Should I have installed my app's module into site-packages first? I'm a bit new to python packaging so maybe this is obvious to others.

Question 3

I got an error the first time I ran this and modified the source to get around it. I was calling the script wrong but I'm including the error here just in case it's a real problem.

lambda-uploader ./
λ Building Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "/Users/agoodnough/src/OVSensorReadingsProcessor/venv/lib/python2.7/site-packages/lambda_uploader/shell.py", line 97, in main
    _execute(args)
  File "/Users/agoodnough/src/OVSensorReadingsProcessor/venv/lib/python2.7/site-packages/lambda_uploader/shell.py", line 52, in _execute
    pkg = package.build_package(pth, cfg.requirements)
  File "/Users/agoodnough/src/OVSensorReadingsProcessor/venv/lib/python2.7/site-packages/lambda_uploader/package.py", line 35, in build_package
    pkg.package()
  File "/Users/agoodnough/src/OVSensorReadingsProcessor/venv/lib/python2.7/site-packages/lambda_uploader/package.py", line 105, in package
    self._copy_src_files(package)
  File "/Users/agoodnough/src/OVSensorReadingsProcessor/venv/lib/python2.7/site-packages/lambda_uploader/package.py", line 118, in _copy_src_files
    os.mkdir(fpath)
OSError: [Errno 17] File exists: '/Users/agoodnough/src/OVSensorReadingsProcessor/.lambda_package/lambda_package/refs'

I've got a patch I could submit if it's useful. https://github.com/agoodno/lambda-uploader/commit/d5b94c11be3e929cf649d5ea0ad3cad329aa41c4. I haven't seen it again now that I'm calling the script with my dist directory instead.

Thanks,

Andy

Ugly crash if lambda.json file has syntax error

My first attempt, I deleted the vpc section of lambda.json, and forgot to remove the trailing comma before it. This yielded some cryptic errors not managed well for a newbie (like myself).
I'm new to gitHub, but I will try to grab the repo, and see if I can't fix this in the coming weeks (if it's not otherwise fixed). To see if I can contribute to the effort.

Just starting up the Lambda learning curve, Thinking it's a big lever to get work done.

Allow a virtualenv for another platform to be included

Instead of building a virtualenv with requirements.txt, allow an existing virtualenv to be used. This would allow us to build a virtualenv for Amazon Linux, and then still deploy lambda functions using lambda-uploader from client machines or other automation that may not be the same architecture as what Lambda runs on.

This is needed for me due to PyPi's cryptography package using different shared libraries on Ubuntu (my desktop) and Amazon Linux (where the function will run).

Missing lib64

According to this tutorial from AWS, we're missing the lib64 directory from being included in packages:

Add the contents of lib and lib64 site-packages to your .zip file.
$ cd  $VIRTUAL_ENV/lib/python2.7/site-packages
$ zip -r9 ~/lambda_function.zip * 
$ cd $VIRTUAL_ENV/lib64/python2.7/site-packages
$ zip -r9 ~/lambda_function.zip *

We should probably be including lib64/python2.7/site-packages in the uploaded lambda function package, in addition to what we're including now.

Add ability to upload to an S3 bucket

The specific need here is if I am using the same package for multiple Lambda functions and I don't want to re-upload for n number of functions. So instead using an s3 bucket we can package and upload once, then just create or update the lambda functions.

_python_executable in package.py doesn't check for None

@horacio3 is seeing the following stack trace:

⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "build\bdist.win32\egg\lambda_uploader\shell.py", line 173, in main
    _execute(args)
  File "build\bdist.win32\egg\lambda_uploader\shell.py", line 73, in _execute
    venv, cfg.ignore, extra_files)
  File "build\bdist.win32\egg\lambda_uploader\package.py", line 47, in build_package
    pkg.build(ignore)
  File "build\bdist.win32\egg\lambda_uploader\package.py", line 68, in build
    self.install_dependencies()
  File "build\bdist.win32\egg\lambda_uploader\package.py", line 140, in install_dependencies
    self._build_new_virtualenv()
  File "build\bdist.win32\egg\lambda_uploader\package.py", line 167, in _build_new_virtualenv
    self._pkg_venv], stdout=PIPE, stderr=PIPE)
  File "C:\Python27\lib\subprocess.py", line 710, in __init__
    errread, errwrite)
  File "C:\Python27\lib\subprocess.py", line 913, in _execute_child
    args = list2cmdline(args)
  File "C:\Python27\lib\subprocess.py", line 616, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'NoneType' is not iterable

It boils down to _find_executable not handling None returned from find_executable:

def _python_executable():
        python_exe = find_executable('python2')
        if python_exe is '':
            python_exe = find_executable('python')

        if python_exe is '':
            raise Exception('Unable to locate python executable')

        return python_exe

Example of find_executable output on Horacio's box:

from distutils.spawn import find_executable
>>> print(find_executable('python2'))
None
>>> print(find_executable('python'))
C:\Python27\python.exe

So then None is returned from _python_executable and this line barfs when you try to pass None as an argument to Popen.

File copy isn't recursing into directories

When copying the files to prep the zip package, the lambda-uploader does not appear to be creating the new parent directories of the files and instead copying the files to the top level directory.

For example '/foo/bar.txt' would show up in the zip '/bar.txt'.

This seems like it might be the same issue as the one described in #9

JSON Serialization or MemoryError

So, I got another error. I have two near-identical functions with different lambda.json's. One works, one does not.

Working:

{
"name": "integrationFunctionName",
"description": "Description. Documentation: https://web.site",
"region": "us-west-2",
"handler": "integration.lambda_handler",
"role": "arn:aws:iam::1234:role/app-int-lambda",
"requirements": ["selenium", "pillow", "pynamodb"],
"ignore": [
".git",
"/*.pyc"
],
"timeout": 30,
"memory": 512
}

This one uploads and functions as expected.

Non-working:

{
"name": "functionName",
"description": "Description. Documentation: https://web.site",
"region": "us-west-2",
"handler": "production.lambda_handler",
"role": "arn:aws:iam::1234:role/app-prod-lambda",
"requirements": ["selenium", "pillow", "pynamodb"],
"ignore": [
".git",
"/*.pyc"
],
"timeout": 30,
"memory": 512
}

This one errors out right after the upload starts.

From lambda-uploader -VV

DEBUG:botocore.auth:Signature:
91f37147018cbb1bc394be46317c60593960212b809510e4f800fee436a70538
DEBUG:botocore.endpoint:Sending http request: <PreparedRequest [GET]>
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): lambda.us-west-2.amazonaws.com
DEBUG:botocore.vendored.requests.packages.urllib3.connectionpool:"GET /2015-03-31/functions/functionName/configuration HTTP/1.1" 404 106
DEBUG:botocore.parsers:Response headers: {'x-amzn-requestid': '8e7d7e55-c6d9-11e5-9312-bfc6b2d91d64', 'content-length': '106', 'connection': 'keep-alive', 'date': 'Fri, 29 Jan 2016 22:42:20 GMT', 'content-type': 'application/json', 'x-amzn-errortype': 'ResourceNotFoundException:http://internal.amazon.com/coral/com.amazonaws.awsgirapi/'}
DEBUG:botocore.parsers:Response body:
{"Message":"Function not found: arn:aws:lambda:us-west-2:1234:function:functionName","Type":"User"}
DEBUG:botocore.hooks:Event needs-retry.lambda.GetFunctionConfiguration: calling handler <botocore.retryhandler.RetryHandler object at 0x7efe3cc94910>
DEBUG:botocore.retryhandler:No retry needed.
DEBUG:lambda_uploader.uploader:function not found creating new function
DEBUG:lambda_uploader.uploader:running create_function_code
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 151, in main
_execute(args)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 82, in _execute
upldr.upload(pkg)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 112, in upload
self.version = self.upload_new(pkg)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/uploader.py", line 89, in upload_new
Publish=self._config.publish,
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 382, in _make_api_call
api_params, operation_model)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 423, in _convert_to_request_dict
api_params, operation_model)
File "/usr/lib/python2.7/dist-packages/botocore/validate.py", line 275, in serialize_to_request
operation_model)
File "/usr/lib/python2.7/dist-packages/botocore/serialize.py", line 412, in serialize_to_request
serialized, shape, shape_members)
File "/usr/lib/python2.7/dist-packages/botocore/serialize.py", line 457, in _serialize_payload
partitioned['body_kwargs'], shape)
File "/usr/lib/python2.7/dist-packages/botocore/serialize.py", line 520, in _serialize_body_params
return json.dumps(serialized_body).encode(self.DEFAULT_ENCODING)
MemoryError

add option to update only lambda configurations

This is a feature request.

I want options to update only lambda configurations.

In lambda-uploader pipeline, there are roughly 3 steps

1. build code and make zip
2. remove working directory
3a. update code
3b. update configuration 

There are options

  • to skip #2 --no-clean
  • to skip #3 --no-upload

And I want to skip #1, #2, #3a to only update configurations.
Some folks might want to just upload an existing Zip file. In this case, skip #1, #2, #3b

So I propose to add options

  • to skip #1(--no-build?)
  • to skip #3a(--no-zip-upload?)
  • to skip #3b(--no-conf-update?)

Error when building example project

When running lambda-upload on the example project, the follow exception occurs:

λ Building Package
λ Uploading Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/lambda_uploader/shell.py", line 163, in main
    _execute(args)
  File "build/bdist.linux-x86_64/egg/lambda_uploader/shell.py", line 88, in _execute
    upldr.upload(pkg)
  File "build/bdist.linux-x86_64/egg/lambda_uploader/uploader.py", line 117, in upload
    self.version = self.upload_new(pkg)
  File "build/bdist.linux-x86_64/egg/lambda_uploader/uploader.py", line 80, in upload_new
    self._validate_package_size(pkg.zipfile)
AttributeError: 'Package' object has no attribute 'zipfile'

VirtualEnv not Included

I have two EC2's setup near-identically and am working with the same code in each. In the first one lambda-uploader works as expected. In the second one lambda-uploader does not include the virtualenv with all those library subfolders in the zip file, only lambda.json and function.py. Both function.py and lambda.json are identical in each. I am not using any command line options with either system.

In the system it doesn't work on running with the -VV arguement displays no errors and it even talks about the very libraries that it doesn't include in the zip.

Any idea what could be causing this? Here is the full -VV output (using --no-upload only so I could examine the zip, which is how I discovered the missing libraries):

lambda-uploader --no-upload -VV
λ Building Package
INFO:lambda_uploader.package:Building new virtualenv and installing requirements
DEBUG:lambda_uploader.package:Virtualenv stdout: New python executable in /home/rdavis/status/.lamba_uploader_temp/venv/bin/python
Installing setuptools, pip, wheel...done.
DEBUG:lambda_uploader.package:Virtualenv stderr:
DEBUG:lambda_uploader.package:Installing requirements found [u'pynamodb'] in config
DEBUG:lambda_uploader.package:Pip stdout: Collecting pynamodb
Using cached pynamodb-1.4.4-py2.py3-none-any.whl
Collecting six (from pynamodb)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting Delorean (from pynamodb)
Collecting botocore>=1.0.0 (from pynamodb)
Using cached botocore-1.3.22-py2.py3-none-any.whl
Collecting python-dateutil>=2.4.2 (from Delorean->pynamodb)
Using cached python_dateutil-2.4.2-py2.py3-none-any.whl
Collecting tzlocal>=1.2 (from Delorean->pynamodb)
Collecting pytz>=2015.7 (from Delorean->pynamodb)
Using cached pytz-2015.7-py2.py3-none-any.whl
Collecting babel>=2.1.1 (from Delorean->pynamodb)
Using cached Babel-2.2.0-py2.py3-none-any.whl
Collecting humanize>=0.5.1 (from Delorean->pynamodb)
Collecting jmespath<1.0.0,>=0.7.1 (from botocore>=1.0.0->pynamodb)
Using cached jmespath-0.9.0-py2.py3-none-any.whl
Collecting docutils>=0.10 (from botocore>=1.0.0->pynamodb)
Installing collected packages: six, python-dateutil, pytz, tzlocal, babel, humanize, Delorean, jmespath, docutils, botocore, pynamodb
Successfully installed Delorean-0.6.0 babel-2.2.0 botocore-1.3.22 docutils-0.12 humanize-0.5.1 jmespath-0.9.0 pynamodb-1.4.4 python-dateutil-2.4.2 pytz-2015.7 six-1.10.0 tzlocal-1.2
DEBUG:lambda_uploader.package:Pip stderr:
INFO:lambda_uploader.package:Copying site packages
INFO:lambda_uploader.utils:Copying source files
INFO:lambda_uploader.package:Copying lib64 site packages
INFO:lambda_uploader.utils:Copying source files
INFO:lambda_uploader.utils:Copying source files
DEBUG:lambda_uploader.utils:Copying /home/rdavis/status/function.py to /home/rdavis/status/.lamba_uploader_temp/lambda_package/
DEBUG:lambda_uploader.utils:Copying /home/rdavis/status/lambda.json to /home/rdavis/status/.lamba_uploader_temp/lambda_package/
INFO:lambda_uploader.package:Creating zipfile
DEBUG:lambda_uploader.package:Zipping /home/rdavis/status/.lamba_uploader_temp/lambda_package/function.py as function.py
DEBUG:lambda_uploader.package:Zipping /home/rdavis/status/.lamba_uploader_temp/lambda_package/lambda.json as lambda.json
λ Fin

botocore-1.3.30 Dependency

The most recent version of awscli (1.10.15) depends on botocore-1.4.6, but this breaks lambda-uploader, which has a dependency on botocore-1.3.30

requirements.txt not found

When no requirements are provided, none are used. The problem is in Package.requirements() because when requires is None (as it is when nothing is provided), the default self._requirements_file is also set to None

Python 3 support

If possible (dependencies allowing) this should support Python 3.

UTF-8 Error

Once I had lambda-uploader working I wrote a little utility script to copy my function.py to a '.λ' directory with its lambda.json. Basically, I wanted to be able to have the same code and do:

λscript --production
Are you sure?! (yes/no)

λscript
(that would just upload integration by default)

Everything worked fine when I named the directory '.1.' If I named it '.λ' I get:

λ Building Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 151, in main
_execute(args)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/shell.py", line 64, in _execute
venv, cfg.ignore)
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/package.py", line 41, in build_package
pkg.prepare_virtualenv()
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/package.py", line 86, in prepare_virtualenv
self.build_new_virtualenv()
File "/usr/local/lib/python2.7/site-packages/lambda_uploader/package.py", line 112, in build_new_virtualenv
raise Exception('virtualenv returned unsuccessfully')
Exception: virtualenv returned unsuccessfully

Ability to specify multiple lambda configs

This is helpful if you want to create multiple lambda functions off of the same code base right now I think it is best to let -c take a list of lambda configs.

Ex:
lambda-uploader -c myfunction1.json, myfunction2.json

Failing on bad execv() argument?

I'm getting this traceback:

lambda-uploader --profile lambda-ops
λ Building Package
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "/Users/mikelloy/Library/Python/2.7/lib/python/site-packages/lambda_uploader/shell.py", line 172, in main
    _execute(args)
  File "/Users/mikelloy/Library/Python/2.7/lib/python/site-packages/lambda_uploader/shell.py", line 73, in _execute
    venv, cfg.ignore, extra_files)
  File "/Users/mikelloy/Library/Python/2.7/lib/python/site-packages/lambda_uploader/package.py", line 47, in build_package
    pkg.build(ignore)
  File "/Users/mikelloy/Library/Python/2.7/lib/python/site-packages/lambda_uploader/package.py", line 68, in build
    self.install_dependencies()
  File "/Users/mikelloy/Library/Python/2.7/lib/python/site-packages/lambda_uploader/package.py", line 138, in install_dependencies
    self._build_new_virtualenv()
  File "/Users/mikelloy/Library/Python/2.7/lib/python/site-packages/lambda_uploader/package.py", line 165, in _build_new_virtualenv
    self._pkg_venv], stdout=PIPE, stderr=PIPE)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
TypeError: execv() arg 2 must contain only strings

Any ideas?

Linked Library Issues

Noticed an issue with modules with linked libraries(PyCrypto for instance), not even using your uploader but I suspect same issues would be present and it might be something cool for you guys to try and solve. If you run this uploader or pip on local OSX box and try and use those dependencies on Lambda it will fail as the ELFs are not linux compatible.

Allow ignoring of library files too

First, thanks for a great project, been very useful.

Proposal

For small lambdas in particular, it would be useful to either:

  • allow a mode that auto-ignores distribution / library files (e.g. --minimal or something) that are generally not needed (see below)
  • or allow the filters specified in lambda.json to apply not just to the source but to the virtualenv-related inclusions too, before packaging into a zip.

Naturally this should be optional as virtualenv is necessary if you're not running on the standard Amazon setup, etc.

Example

A small python lambda function, consisting of a few source files was uploading very quickly.

By adding a single pip requirement (also, as it happens, very small), and rebuilding, the size of the upload suddenly shot up by ~1000% (many megabytes) - on further examination this was due to the inclusion in the zip of:

  • Compiled versions of all library files (*.pyc)
  • Various Windows binaries e.g. gui.exe which will never be useful on AMIs
  • setuptools, wheel, pip etc and their tests
  • CA Certs (357k) - pulled in by requests (for pip)
    etc

Thanks!

Cannot upload due to git pack file being locked

I'm unable to build because a file in .lambda_uploader_temp cannot be deleted:

$ lambda-uploader.exe --config get.json
⁉️ Unexpected error. Please report this traceback.
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\lambda_uploader\shell.py", line 147, in main
    _execute(args)
  File "c:\python27\lib\site-packages\lambda_uploader\shell.py", line 64, in _execute
    venv, cfg.ignore)
  File "c:\python27\lib\site-packages\lambda_uploader\package.py", line 32, in build_package
    pkg.clean_workspace()
  File "c:\python27\lib\site-packages\lambda_uploader\package.py", line 51, in clean_workspace
    shutil.rmtree(self._temp_workspace)
  File "c:\python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "c:\python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "c:\python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "c:\python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "c:\python27\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "c:\python27\lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 5] Access is denied: 'D:\\lambda\\python\\facility-service\\.lamba_uploader_temp\\lambda_package\\.git\\objects\\pack\\pack-22f84b49ef7d82978a5df62494ba39b3aa0147a4.idx'

Any idea what's going on? I tried blasting the folder and recloning my repo, but got the same result. Interestingly, it does not happen on my Mac, only on Windows. I also ran handle.exe looking for handles on this path, but there were none, which suggests the lock is in the Python process.

Also, using the switch to skip cleanup doesn't skip cleanup (see the Package class, build fcn).

Support subscription to SNS Topic and CloudWatch Event Rule

It would be nice if the lambda config could be extended to automatically register the Lambda with a SNS Topic, CloudWatch Event Rule, etc.

This would add two properties (events and topics) to the root JSON object, both of which should be lists:

{
  "name": "Example_Lambda",
  "description": "This is a lambda",
  "region": "us-west-2",
  "handler": "function.lambda_handler",
  "role": "arn:aws:iam::0123456789012:role/LambdaRole",
  "timeout": 30,
  "memory": 128,
  "ignore": [
    "/*.pyc",
    "lambda.json"
  ],
  "events": [
    {
      "State": "ENABLED",
      "EventPattern": "{\"detail-type\":[\"AWS API Call via CloudTrail\"],\"detail\":{\"eventSource\":[\"ec2.amazonaws.com\"],\"eventName\":[\"StartInstances\",\"RunInstances\",\"RunScheduledInstances\"]}}",
      "Name": "Example-Lambda-Trigger",
      "Description": "Triggers lambda execution"
    }
  ],
  "topics": [
    "arn:aws:sns:us-west-2:0123456789012:Example-Lambda-Topic"
  ]
}

For both use cases, the object should be created if it does not exist, and permissions should be granted as necessary to allow invocation of the lambda by the calling service.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.