logandk / serverless-wsgi Goto Github PK

Using django-rest-framework-json-api, all of the responses are unnecessarily base64 encoded. The fix is simple. In wsgi.py, add application/vnd.api+json to TEXT_MIME_TYPES.

Thank you.

I'm trying to use environment variables to access CF stack outputs and the following works for me in Lambda:

  environment:
    VAR1:
      Fn::ImportValue: "UserPool::Id"
    # OR
    VAR2:
      Ref: UserPool

But it won't work for sls wsgi serve:

>>> os.environ['VAR1']
[object Object]
>>> os.environ['VAR2']
[object Object]

I don't know much about how serverless plugins work but my best guess is that it comes from here:
https://github.com/logandk/serverless-wsgi/blob/master/index.js#L163

Is this a serverless problem? or would it be possible for serverless-wsgi to store the resolved Fn::ImportValues on sls deploy and reproduce them later? or pull from cloudformation?

Hi,

I am trying to deploy a django graphql api to aws. I have exactly followed this example. In addition I have added the aws endpoint given after deploying to aws in allowed_hosts of the settings.py. Anyway, I always get "{"message": "Internal server error"}" independent of which view I am trying to access. According to serverless-wsgi the app under custom should be set to "cookbook.wsgi.application". But do I need to adjust the prefix of wsgi.handler under functions? Here it is "api", whereas in the official serverless blog post "Build a Python REST API with Serverless, Lambda, and DynamoDB" it is "app". Both are flask, to what should it be adjusted in this django case?

Currently my serverless.yml looks like this:

service: cookbook

plugins:
  - serverless-python-requirements
  - serverless-wsgi

custom:
  wsgi:
    app: cookbook.wsgi.application
    packRequirements: false
  pythonRequirements:
    dockerizePip: non-linux

provider:
  name: aws
  runtime: python3.6
  stage: dev
  region: eu-central-1

functions:
  app:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: 'ANY {proxy+}'

Cheers

I am running under ubuntu 16.04. This has python3 installed but no python. python(2) is an optional package, and the system doesn't install it by default. I'd like to keep my system python2-free.

I am trying the sample flask app, and only slightly modified serverless.yml:

service: my-flask

provider:
  name: aws
  runtime: python3.6
  stage: dev
  region: eu-west-1

plugins:
  - serverless-wsgi

functions:
  api:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: ANY {proxy+}

custom:
  wsgi:
    app: api.app

When I try to deploy I get the following error:

$ serverless deploy -v
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...

  Error --------------------------------------------------

  spawnSync python ENOENT

There are two places in index.js which have child_process.spawnSync('python', ... so I changed them both to python3

Now I get a different error:

  Error --------------------------------------------------

  Unable to load virtualenv, please install

Fixed by apt-get install python3-virtualenv

But now it fails, and the error message is completely blank :-(

Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...

  Error --------------------------------------------------


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Setting SLS_DEBUG=1 I get:

Error
    at module.exports.logError.errorHandlingError (/usr/lib/node_modules/serverless/lib/classes/Error.js:94:11)
    at initializeErrorReporter.then.catch.e (/usr/lib/node_modules/serverless/bin/serverless:42:3)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at __dirname (/usr/lib/node_modules/serverless/bin/serverless:40:9)
    at Object.<anonymous> (/usr/lib/node_modules/serverless/bin/serverless:43:4)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:389:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:504:3

Still no good. That seems like a bug in its own right - error messages shouldn't be discarded!

So I bring strace to bear:

strace -s 128 -f serverless deploy -v
...

and now I see the error:

[pid  5633] read(21, "The executable python2 (from --python=python2) does not exist\n", 65536) = 62

Using virtualenv in Ubuntu requires virtualenv -p python3

Modified requirements.py:

    sys.argv = ['', venv_dir, '--quiet', '-p', 'python3']

After this, deployment works, phew.

I'm not sure of the right clean fix. Maybe there should be a setting likecustom.wsgi.python: python3 which overrides both the binary and adds the -p option to virtualenv?

My requirements.txt is frozen, and I don't want the pip install to pull some python 2 dependencies (my lambda runtime is python3).

Concrete example:

(make new virtualenv)
$ pip-install python-jose
$ pip freeze > requirements.txt 
$ cat requirements.txt 
ecdsa==0.13
future==0.17.1 <<<<<<<<< remove this line (python2 only, size is 1.5 megs)
pyasn1==0.4.4
python-jose==3.0.1
rsa==4.0
six==1.11.0

This is a matter of keeping the lambda function zip file size down. But if I do this currently, the serverless-wsgi plugin re-pulls the (transitive) future dependency. I don't want that.

A quick test shows that my code would still work without the future dependency.

$  pip uninstall future
Uninstalling future-0.17.1:
(...)
Proceed (y/n)? y
  Successfully uninstalled future-0.17.1
$ python
Python 3.6.6 (default, Aug 26 2018, 13:00:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jose import jwt
>>> token = jwt.encode({'key': 'value'}, 'secret', algorithm='HS256')
>>> print(token)
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJrZXkiOiJ2YWx1ZSJ9.JPIDicqvQ6GAh14yE2yZ3wnZQ0LiLNTTRDtJgLZcn98

Could a capability be added to pass --no-deps to the pip install command-line run in requirements.py? Maybe a way to pass pip command-line arguments through the serverless.yml's custom: section, or some environment variable?

I'll try the python-requirements plugin to see what happens there.

I'm trying to use this on a Django project which doesn't use werkzeug. In order for it to work, werkzeug is a required dependency. Without installing werkzeug I received:

Unable to import module 'wsgi': No module named werkzeug.datastructures

I'm not sure if/how I'm getting this behavior but unless I've added this mime type (application/javascript) here:
https://github.com/logandk/serverless-wsgi/blob/master/wsgi.py#L16

All of my javascript files loading in are base64 encoded and break rendering. I've adding this mime type and everything is fixed. This seems like a bug unless I'm overlooking something in configuration.

Is there a way to run CLI commands on the deployed application? A primary example of where this would be needed would be invoking a "manage" command for django, like collectstatic.
Something like serverless manage "collectstatic --no-input" would be ideal.

As it stands, running commands like this is not possible using serverless-wsgi, or at least isn't clearly defined how to do so.

I have service which has a WSGI application and some regular lambda functions.

The non-WSGI code api.py is this:

# https://github.com/logandk/serverless-wsgi/#automatic-requirement-packaging
import os
import sys
root = os.path.abspath(os.path.join(os.path.dirname(__file__)))
sys.path.insert(0, os.path.join(root, '.requirements'))

import xmltodict

def handler(event, context):
    return xmltodict.unparse({"foo": "BAR"})

serverless.yml:

functions:
  web:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: ANY {proxy+}

  api:
    handler: api.handler

custom:
  wsgi:
    app: app.app

requirements.txt

xmltodict

After a full deploy, the non-WSGI code is fine.

$ sls deploy -v
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (2.06 MB)...
...
$ sls invoke -f api -l
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<foo>BAR</foo>"

However, after redeploying just the function, it fails.

$ sls deploy function -f api
Serverless: Packaging function: api...
Serverless: Excluding development dependencies...
Serverless: Uploading function: api (2.02 KB)...
Serverless: Successfully deployed function: api
$ sls invoke -f api -l
{
    "errorMessage": "Unable to import module 'api'"
}
--------------------------------------------------------------------
START RequestId: 94f3188a-9239-11e7-8d0e-01bfd0695980 Version: $LATEST
Unable to import module 'api': No module named 'xmltodict'

Notice how the sls deploy function only uploads 2.02KB.

Aside: having a service which has both WSGI and non-WSGI components is very useful to me. Use cases:

1. I want to bundle an API (directly called by lambda) together with the web interface for managing the API.
2. I want to standardise on serverless-wsgi and not have to use serverless-python-requirements as well
3. I might want to have a WSGI app which also contains handlers for other AWS events, e.g. S3 buckets

(so it would be nice to have first-class support for such functions, so that they don't need to add .requirements to the path themselves)

When running in python3.7 virtualenv this throws:

spawnSync python3.6 ENOENT

I've directory structure as following

├── app
│   ├── common
│   │   ├── _
│   │   
 |   ├── __init__.py
├── serverless.yml
 

(Some part omitted)

__init__.py contains a global variable app which flask application.

In serverless.yml, I've configured the app as following app

functions:
  api:
    handler: wsgi.handler
    events:
      - http: ANY {proxy+}
custom:
  wsgi:
    app: app.__init__.app

There is one Flask API written as

@app.route('/api/v1/status', methods=['GET'])
def application_status():
    """
    Application status
    :return: success
    """

After deploying, I get the following endpoint

endpoints:
  ANY - https://9vxxx7axx0.execute-api.us-east-1.amazonaws.com/dev/{proxy+}

When I try to send GET request on this endpoint
http://9xxxx7axx0.execute-api.us-east-1.amazonaws.com/dev/api/v1/status

I get the following output:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>

What am I missing?

using 'wsgi serve' definitely helps speed up development, but i'm looking into running the project directly with a line by line debguger (pydev). it appears i'm having an issue due to environment variables not being read in by serverless as they are when using 'wsgi serve'.

not a big problem to overcome, but i thought i'd ask if there are any recommendations for debugging with breakpoints.

I would very much like to use the CodeSha256 attribute to check whether multiple stages are running identical code.

Unfortunately, every time I do sls deploy, even to the same stage, the generated zip file has a slightly different content and therefore different hash.

This turns out to be due to changes in the *.dist-info from pip packages added by serverless-wsgi - details below. Stripping this out makes the sha256 stable.

Since the dist-info data is not required for successful running of the library, please could serverless-wsgi filter it out by default?

Demonstration:

$ sls deploy && cp .serverless/foo.zip a.zip
...
$ sls deploy && cp .serverless/foo.zip b.zip
...
$ ls -l a.zip b.zip
-rw-rw-r-- 1 ubuntu ubuntu 8011038 Jan 16 10:35 a.zip
-rw-rw-r-- 1 ubuntu ubuntu 8011006 Jan 16 10:36 b.zip
$ unzip -v a.zip >a; unzip -v b.zip >b; diff -u a b
--- a	2018-01-16 10:37:11.733316280 +0000
+++ b	2018-01-16 10:37:11.753317413 +0000
@@ -1,4 +1,4 @@
-Archive:  a.zip
+Archive:  b.zip
  Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
 --------  ------  ------- ---- ---------- ----- --------  ----
     9264  Defl:N     3823  59% 1980-01-01 00:00 d15c6bba  api.py
@@ -200,7 +200,7 @@
     1534  Defl:N      819  47% 1980-01-01 00:00 66908bfa  Werkzeug-0.14.1.dist-info/LICENSE.txt
     3872  Defl:N     1672  57% 1980-01-01 00:00 5966375f  Werkzeug-0.14.1.dist-info/METADATA
     1452  Defl:N      659  55% 1980-01-01 00:00 2f6f6a96  Werkzeug-0.14.1.dist-info/metadata.json
-    6775  Defl:N     3018  56% 1980-01-01 00:00 019bf153  Werkzeug-0.14.1.dist-info/RECORD
+    6775  Defl:N     3003  56% 1980-01-01 00:00 4ec41c26  Werkzeug-0.14.1.dist-info/RECORD
        9  Defl:N       11 -22% 1980-01-01 00:00 d4cd6b6c  Werkzeug-0.14.1.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 aa646eec  Werkzeug-0.14.1.dist-info/WHEEL
       63  Defl:N       61   3% 1980-01-01 00:00 8eaf4047  certifi/__init__.py
@@ -213,7 +213,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  certifi-2017.11.5.dist-info/INSTALLER
     2546  Defl:N     1145  55% 1980-01-01 00:00 c2b36ca6  certifi-2017.11.5.dist-info/METADATA
      913  Defl:N      410  55% 1980-01-01 00:00 8a2edf06  certifi-2017.11.5.dist-info/metadata.json
-    1201  Defl:N      688  43% 1980-01-01 00:00 61dc9d14  certifi-2017.11.5.dist-info/RECORD
+    1201  Defl:N      688  43% 1980-01-01 00:00 4755d1e9  certifi-2017.11.5.dist-info/RECORD
        8  Defl:N       10 -25% 1980-01-01 00:00 f6eec75d  certifi-2017.11.5.dist-info/top_level.txt
      113  Defl:N       98  13% 1980-01-01 00:00 7b96df94  certifi-2017.11.5.dist-info/WHEEL
     1731  Defl:N      883  49% 1980-01-01 00:00 21a13c64  certifi-2017.7.27.1.dist-info/DESCRIPTION.rst
@@ -269,7 +269,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  chardet-3.0.4.dist-info/INSTALLER
     3239  Defl:N     1403  57% 1980-01-01 00:00 c8754a44  chardet-3.0.4.dist-info/METADATA
     1375  Defl:N      614  55% 1980-01-01 00:00 e8bec562  chardet-3.0.4.dist-info/metadata.json
-    6167  Defl:N     2631  57% 1980-01-01 00:00 390df0c2  chardet-3.0.4.dist-info/RECORD
+    6167  Defl:N     2623  58% 1980-01-01 00:00 53b58155  chardet-3.0.4.dist-info/RECORD
        8  Defl:N       10 -25% 1980-01-01 00:00 927b2990  chardet-3.0.4.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  chardet-3.0.4.dist-info/WHEEL
       58  Defl:N       51  12% 1980-01-01 00:00 9b5c4700  idna/__init__.py
@@ -284,7 +284,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  idna-2.6.dist-info/INSTALLER
     8866  Defl:N     3640  59% 1980-01-01 00:00 e49d9981  idna-2.6.dist-info/METADATA
     1097  Defl:N      516  53% 1980-01-01 00:00 af1dfd18  idna-2.6.dist-info/metadata.json
-    1484  Defl:N      790  47% 1980-01-01 00:00 a82621a0  idna-2.6.dist-info/RECORD
+    1484  Defl:N      791  47% 1980-01-01 00:00 a025dfb1  idna-2.6.dist-info/RECORD
        5  Defl:N        7 -40% 1980-01-01 00:00 383ed7a7  idna-2.6.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  idna-2.6.dist-info/WHEEL
   101373  Defl:N    33527  67% 1980-01-01 00:00 6291aaa7  pkg_resources/__init__.py
@@ -307,7 +307,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  pkg_resources-0.0.0.dist-info/INSTALLER
      177  Defl:N      118  33% 1980-01-01 00:00 ea4f14c3  pkg_resources-0.0.0.dist-info/METADATA
      221  Defl:N      168  24% 1980-01-01 00:00 35f04cfd  pkg_resources-0.0.0.dist-info/metadata.json
-    2812  Defl:N     1100  61% 1980-01-01 00:00 33baba2b  pkg_resources-0.0.0.dist-info/RECORD
+    2812  Defl:N     1102  61% 1980-01-01 00:00 c845a63d  pkg_resources-0.0.0.dist-info/RECORD
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  pkg_resources-0.0.0.dist-info/WHEEL
    34435  Defl:N     9280  73% 1980-01-01 00:00 a5b6ac27  pytz/__init__.py
     1333  Defl:N      596  55% 1980-01-01 00:00 0047d14f  pytz/exceptions.py
@@ -918,7 +918,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  pytz-2017.2.dist-info/INSTALLER
    20126  Defl:N     7645  62% 1980-01-01 00:00 f0b616fd  pytz-2017.2.dist-info/METADATA
      906  Defl:N      513  43% 1980-01-01 00:00 e0a03182  pytz-2017.2.dist-info/metadata.json
-   52688  Defl:N    21643  59% 1980-01-01 00:00 f69fed32  pytz-2017.2.dist-info/RECORD
+   52688  Defl:N    21644  59% 1980-01-01 00:00 3be77323  pytz-2017.2.dist-info/RECORD
        5  Defl:N        7 -40% 1980-01-01 00:00 4f3a402a  pytz-2017.2.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  pytz-2017.2.dist-info/WHEEL
        1  Defl:N        3 -200% 1980-01-01 00:00 32d70693  pytz-2017.2.dist-info/zip-safe
@@ -944,7 +944,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  requests-2.18.4.dist-info/INSTALLER
    50496  Defl:N    20134  60% 1980-01-01 00:00 247dbf69  requests-2.18.4.dist-info/METADATA
     1638  Defl:N      727  56% 1980-01-01 00:00 38843345  requests-2.18.4.dist-info/metadata.json
-    2849  Defl:N     1312  54% 1980-01-01 00:00 19a41c86  requests-2.18.4.dist-info/RECORD
+    2849  Defl:N     1313  54% 1980-01-01 00:00 218d4c1f  requests-2.18.4.dist-info/RECORD
        9  Defl:N       11 -22% 1980-01-01 00:00 2ec0e227  requests-2.18.4.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  requests-2.18.4.dist-info/WHEEL
     2853  Defl:N     1195  58% 1980-01-01 00:00 87f53e92  urllib3/__init__.py
@@ -987,7 +987,7 @@
        4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  urllib3-1.22.dist-info/INSTALLER
    32027  Defl:N    13222  59% 1980-01-01 00:00 d06cf4de  urllib3-1.22.dist-info/METADATA
     1653  Defl:N      718  57% 1980-01-01 00:00 4cb24210  urllib3-1.22.dist-info/metadata.json
-    5656  Defl:N     2306  59% 1980-01-01 00:00 8d6f40f4  urllib3-1.22.dist-info/RECORD
+    5656  Defl:N     2292  60% 1980-01-01 00:00 702cfcc1  urllib3-1.22.dist-info/RECORD
        8  Defl:N       10 -25% 1980-01-01 00:00 c052ff1b  urllib3-1.22.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  urllib3-1.22.dist-info/WHEEL
     6842  Defl:N     2579  62% 1980-01-01 00:00 5b85439b  werkzeug/__init__.py
@@ -1047,4 +1047,4 @@
       10  Defl:N       12 -20% 1980-01-01 00:00 06727e3b  xmltodict-0.11.0.dist-info/top_level.txt
      110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  xmltodict-0.11.0.dist-info/WHEEL
 --------          -------  ---                            -------
-17651153          7855116  56%                            1045 files
+17651153          7855084  56%                            1045 files

And I can confirm that stripping out the dist-info makes the sha256 identical:

$ sha256sum a.zip b.zip
2d8719ad8b94d84f6535209a494a6608af333e67728163dd53ba97f1fcfa07b3  a.zip
61a795ad7c3659e3992f6c71f00c48eb9ae50849809b7a2c181ec6c880852664  b.zip
$ zip -d a.zip '*.dist-info/*'
...
$ zip -d b.zip '*.dist-info/*'
...
$ sha256sum a.zip b.zip
ad846ef5c8fd7d480e273d33f4ef1e2e5d09651658b54e9c80b32b89e3486702  a.zip
ad846ef5c8fd7d480e273d33f4ef1e2e5d09651658b54e9c80b32b89e3486702  b.zip

I am using the serverless-wsgi with a Django Rest Framework project.

Using the browsable API functionality of DRF, when I run the local server using sls wsgi serve -p 8000, I get loads of 404 errors for the rest framework's static files. When I run the local server using ./manage.py runserver, I get no 404 errors.

EDIT: Not a big deal, since I am not going to use the browsable api functionality in production anyways, but it's great for testing locally

See the contrasting screenshots below

Changes from #33 don't appear to work with ubuntu 16.04. Reproducing case:

serverless.yml

service: my-service

provider:
  name: aws
  runtime: python3.6
  stage: dev
  region: eu-west-1

package:
  exclude:
    - "**"
  include:
    - handler.py

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: /foo
          method: get

plugins:
  - serverless-wsgi

custom:
  wsgi:
    app: handler.hello
    pythonBin: python3

handler.py

import xmltodict

def hello(event, context):
    body = {
        "message": "Hello lambda world",
        "input": event
    }

    response = {
        "statusCode": 200,
        "body": xmltodict.unparse({"result": body}),
    }

    return response

requirements.txt

xmltodict

Results:

• When the function is called, it returns {"message": "Internal server error"}.
• CloudWatch logs show: Unable to import module 'handler': No module named 'xmltodict'
• .serverless/my-service.zip is huge (20.52MB) and contains a load of unrelated junk in a .venv top-level directory.

$ unzip -v .serverless/my-service.zip
Archive:  .serverless/my-service.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
     241  Defl:N      152  37% 1980-01-01 00:00 4c3917f2  handler.py
    6091  Defl:N     2497  59% 1980-01-01 00:00 04039966  wsgi.py
      13  Defl:N       15 -15% 1980-01-01 00:00 a6e25e52  .wsgi_app
    2108  Defl:N      833  61% 1980-01-01 00:00 84ee13bb  .venv/bin/activate
    1137  Defl:N      537  53% 1980-01-01 00:00 078145a3  .venv/bin/activate_this.py
    1050  Defl:N      532  49% 1980-01-01 00:00 6824902d  .venv/bin/activate.csh
    2248  Defl:N      900  60% 1980-01-01 00:00 82d1587f  .venv/bin/activate.fish
     278  Defl:N      206  26% 1980-01-01 00:00 514e61e4  .venv/bin/easy_install
     278  Defl:N      206  26% 1980-01-01 00:00 514e61e4  .venv/bin/easy_install-2.7
     250  Defl:N      188  25% 1980-01-01 00:00 e0fcf0e5  .venv/bin/pip
     250  Defl:N      188  25% 1980-01-01 00:00 e0fcf0e5  .venv/bin/pip2
     250  Defl:N      188  25% 1980-01-01 00:00 e0fcf0e5  .venv/bin/pip2.7
 5429482  Defl:N  1653333  70% 1980-01-01 00:00 46c4a7b2  .venv/bin/python
    2367  Defl:N      905  62% 1980-01-01 00:00 9b7d9935  .venv/bin/python-config
 5429482  Defl:N  1653333  70% 1980-01-01 00:00 46c4a7b2  .venv/bin/python2
 5429482  Defl:N  1653333  70% 1980-01-01 00:00 46c4a7b2  .venv/bin/python2.7
     257  Defl:N      193  25% 1980-01-01 00:00 8fcca9bf  .venv/bin/wheel
   18619  Defl:N     5186  72% 1980-01-01 00:00 b0f6c3cf  .venv/lib/python2.7/_abcoll.py
   30978  Defl:N     8802  72% 1980-01-01 00:00 5f62d05f  .venv/lib/python2.7/_abcoll.pyc
    5911  Defl:N     1619  73% 1980-01-01 00:00 0d0944be  .venv/lib/python2.7/_weakrefset.py
   12100  Defl:N     2989  75% 1980-01-01 00:00 bbd04380  .venv/lib/python2.7/_weakrefset.pyc
    7145  Defl:N     2393  67% 1980-01-01 00:00 c9f117b3  .venv/lib/python2.7/abc.py
    6780  Defl:N     2784  59% 1980-01-01 00:00 8d2304aa  .venv/lib/python2.7/abc.pyc
...
  151008  Defl:N    72686  52% 1980-01-01 00:00 9d6eb4c8  .venv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl
  118619  Defl:N    56392  53% 1980-01-01 00:00 169d61c0  .venv/share/python-wheels/wheel-0.29.0-py2.py3-none-any.whl
--------          -------  ---                            -------
67493747         21060069  69%                            2104 files

Note that the .venv subdirectory appears to include a bunch of python2 stuff, even though I have specified pythonBin: python3.

And it doesn't include the package from requirements.txt

$ unzip -v .serverless/my-service.zip | grep xmltodict
$ 

If an app has gained a top-level __pycache__ directory (e.g. because you ran the app locally to test it) then this causes sls deploy to fail:

$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Linking required Python packages...

  Serverless Error ---------------------------------------

  Unable to link dependency '__pycache__' because a file by the same name exists in this service

Workaround is to rm -rf __pycache__ first.

(Aside: I wonder if the requirements bundle should also exclude *.pyc files, so they are rebuilt server-side when the application starts up)

I'm getting the following error when I call "/" for the simple test app below

{"message":"Missing Authentication Token"}

api.py

from flask import Flask
app = Flask(__name__)

@app.route("/")
def index():
    return "INDEX"

@app.route("/cats")
def cats():
    return "Cats"

@app.route("/dogs/<id>")
def dog(id):
    return "Dog = " + str(id)

serverless.yml

service: example

provider:
  name: aws
  runtime: python2.7

plugins:
  - serverless-wsgi

functions:
  api:
    handler: wsgi.handler
    events:
      - http: ANY {proxy+}

custom:
  wsgi:
    app: api.app

Calls to /cats and /dogs/x work fine. Oddly, calling / from the API Gateway console returns successfully, but not when calling from curl or a web browser.

I following the tutorial here: https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/:

It was working wonderfully for me, but I've come back to it a few weeks later (all in git, so no changes) and I'm now getting the following error:

Traceback (most recent call last):
  File "/Users/aloughran/Code/node_modules/serverless-wsgi/serve.py", line 50, in <module>
    serve(*sys.argv[1:])
  File "/Users/aloughran/Code/node_modules/serverless-wsgi/serve.py", line 42, in serve
    use_evalex=True)
  File "/Library/Python/2.7/site-packages/werkzeug/serving.py", line 720, in run_simple
    s.bind((hostname, port))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known.

The problem is more than likely an issue with my configuration, but I'm lost on working out where the nodename/servname should be passed by the wsgi config, or what could be tripping me up.

Thanks in advance.

When I have unicode characters in my paylaod the CONTENT_LENGTH seems to be different than the actual wsgi.input length that I'm sending to my flask app

input: '{"test":"é"}'
len: 12 (should be 13, afaik)

Hi ,

Following by the blog post, https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/#converting-an-existing-flask-application.

Just wondering how can I get the lambda event and context ? can I just print event as below?
tried but it did not work

I would appreciate it if anyone can help.

# app.py
import os
import boto3

from flask import Flask, jsonify, request
app = Flask(__name__)

event = os.environ['event']
print event

if IS_OFFLINE:
    client = boto3.client(
        'dynamodb',
        region_name='localhost',
        endpoint_url='http://localhost:8000'
    )
else:
    client = boto3.client('dynamodb')

@app.route("/")
def hello():
    return jsonify(os.environ)
... rest of application code ...

I am trying to setup a flask app using serverless-wsgi plugin on aws lambda. I am using api gateway proxy integration and have also enabled this content type in Binary Support. All my POST request's are getting through except for the ones with content-type multipart/form-data.

Does your plugin support Binary data via API gateway? If it does how would I get the contents of my file using flask request module:
Normally I can use below to save a file locally on my server:

data = request.files['customer_speech_file']
data.save('/tmp/' + str(data.filename))

I'm not certain of the best way to debug this problem, but my WSGI app (Falcon, in this case) sets multiple Set-Cookie headers on certain responses and the cookies are correctly set when running locally.

However, when running the same code after deploying to Lambda (using serverless-wsgi), my application works entirely as expected with the notable exception that all responses only ever contain a single set-cookie header with a single cookie value in them.

This problem is evident in Chrome and Safari (on Mac).

Any chance this could be released as a standalone python module. Looks like it could be really useful outside the serverless framework.

Custom authorizers have a serious flaw in that they can not return headers. This means that the CORS header can not be set for 401's.

To work around this I will "allow" every request, but include the authorization status and reason(missing token, invalid token, expired, etc) in the authorizer context. However serverless-wsgi does not currently extract those keys.

I would like to setup a way to extract these authorizer keys...

When I run serverless wsgi serve I'm having problems with the requests taking a long time (10seconds - 1min) to return a response. Sometimes the response will not return at all. To avoid hitting a database I'm sending GET request to a route that immediately returns 'hello world' 200.

So far the only predictable behaviour I can find is that making a request using jquery will always cause a long delay before returning. If I try to request the same URL in the browser tab (whilst the jquery request is still waiting) the tab also waits a long time for for the request to return.

However, once the jquery request returns, calling the URL in a browser tab then returns immediately. Refreshing the tab gives a response immediately.

Sometimes, if jquery request is pending, calling the URL in a browser tab WILL return immediately. When this happens, this seem to also make the jquery request return a response.

Jquery code below:

      $( document ).ready(function() {
        $.get("http://localhost:5000/test", function(data, status){
          alert("Data: " + data + "\nStatus: " + status);
        });
      });

I've tried restarting the server, browsers and my machine, but the problem still persists.

I've tried multiple browsers (safari, chrome, firefox) and all have the same issue.

I've tried making a javascript request using axios instead of jquery but still the requests are returning slowly.

Does anyone have any idea how I might solve this or any additional debugging I can do?

I'm using:

• serverless-wsgi 1.4.9
• node 5.6.0
• macOS 10.12

Serverless: WARNING: Plugin ServerlessWSGI uses deprecated hook before:deploy:createDeploymentArtifacts,
                     use package:createDeploymentArtifacts hook instead

serverless-wsgi version 1.3.0, serverless 1.23.0

Use case: Django on AWS

Any attempt to access the POST body with serverless-wsgi deployed on AWS results in Django trying to concatenate a bytes and a str - for example, the Django admin page:

Environment:


Request Method: POST
Request URL: https://<redacted>.execute-api.us-east-1.amazonaws.com/dev/admin/login/?next=%2Fdev%2Fadmin%2F

Django Version: 1.11.4
Python Version: 3.6.1
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'rest_framework',
 'rest_framework.authtoken']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/var/task/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/var/task/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)

File "/var/task/django/middleware/csrf.py" in process_view
  298.                     request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')

File "/var/task/django/core/handlers/wsgi.py" in _get_post
  132.             self._load_post_and_files()

File "/var/task/django/http/request.py" in _load_post_and_files
  311.             self._post, self._files = QueryDict(self.body, encoding=self._encoding), MultiValueDict()

File "/var/task/django/http/request.py" in body
  272.                 self._body = self.read()

File "/var/task/django/http/request.py" in read
  331.             return self._stream.read(*args, **kwargs)

File "/var/task/django/core/handlers/wsgi.py" in read
  50.             result = self.buffer + read_data

Exception Type: TypeError at /admin/login/
Exception Value: can't concat bytes to str

The str comes from serverless-wsgi, which uses a StringIO to pass wsgi.input to the wsgi app. This should be fixed to use a BytesIO in Python 3, as Zappa has done:

Miserlou/Zappa@2a856fd

Posting a PR shortly, after I do some testing.

We've run into a similar situation as described in #55 .

We request to please add 'application/problem+json' to the TEXT_MIME_TYPES list in https://github.com/logandk/serverless-wsgi/blob/1.4.7/wsgi.py#L21 .

To add some context, we are using serverless-wsgi with zalando/connexion and the connexion framework often returns mimetype 'appliaction/problem+json' as seen here: https://github.com/zalando/connexion/blob/1.5.2/connexion/problem.py#L39

This appears to trip up serverless-wsgi into incorrectly encoding the response body in base64. Adding the new entry to the TEXT_MIME_TYPES list appears to fix the issue.

Thank you for your time.

I'm using serverless to deploy a Django (REST Framework) API as an AWS Lambda function, and I'm also going to be deploying other Lambda functions that are not WSGI apps. I have this all in one repo right now, with serverless.yml alongside api, where api is the Django project directory.

Thus, I set custom.wsgi.app: api.api.wsgi.application. This allows the WSGI handler to find my app just fine, but when the app runs, it appears that the working directory is still the root of my repo, rather than the root of the Django app, because I get an import error on api.settings. I get the same error message when deploying the endpoint to AWS and testing there.

Sorry if this is misfiled; I'm not sure if this is a question about Django, serverless, or serverless-wsgi, as I'm new to all of them. :) Is there a way to make sure the function that runs my Django app from a working directory other than where serverless.yml lives? Is that up to serverless or serverless-wsgi, or Django?

https://github.com/logandk/serverless-wsgi/blob/master/wsgi.py#L144

This needs to be changed like was done on the Zappa framework:

https://github.com/Miserlou/Zappa/pull/938/files#diff-b4b0b6c0ef632d8dc89e72cca21a5cdeR448

Otherwise, the hidden wrapper function that Lambda uses fails to json.dumps() the object.

Please resolve ASAP. We're blocked using Python 3.6. I'm surprised no one else has complained about this yet.

This stems from issues I was having here in handling multipart/form-data with serverless-wsgi.

I've used the serverless-apigw-binary plugin as recommended by the File Uploads section of the Readme. I've isolated my problem to this line in wsgi.py. Pasting some code in here for context:

75    body = event[u'body'] or ''
76    if event.get('isBase64Encoded', False):
77        body = base64.b64decode(body)
78    body = to_bytes(wsgi_encoding_dance(body))

The body is Base 64 encoded, so it goes through the decoding in line 77.

After the body is base64 encoded but before it goes through the wsgi_encoding_dance and to_bytes functions, the body is a bytes object that is the proper image. If I comment out line 78, it works fine.

I added some logging around line 78, and the bytes object is changed when decoding from bytes to str in the wsgi_encoding_dance() and then the reconversion to bytes in to_bytes. I logged the length before and after the line, and the body went from a length of 145647 to 218875.

I don't know enough about encoding & decoding to fix this myself, so looking for advice. Would it be possible to skip the WSGI encoding dance & to_bytes conversion if the body is already a bytes object? If not, is there some other proxy we can look at to see whether the body needs to go through that conversion?

Let me know if you need any more details 😄

serverless-wsgi currently sets the REMOTE_ADDR variable like this:

        'REMOTE_ADDR':
            headers.get(u'X-Forwarded-For', '').split(', ')[0],

But that's insecure, since anyone can submit a request with their own X-Forwarded-For: header, containing one or more IPs. API gateway will append to the end of this, so the header becomes:

X-Forwarded-For: <spoofed-address(es)>, <real-source>, <api-gateway>

and REMOTE_ADDR will be set to the first spoofed address.

What should be returned is <real-source>, which you can get via:

        'REMOTE_ADDR':
            headers.get(u'X-Forwarded-For', '').split(', ')[-2],

Of course that will fail if for some reason the header is missing or broken, so safer would be:

ips = headers.get(u'X-Forwarded-For', '').replace(' ','').split(',')
if len(ips) == 1:
    remote_addr = ips[0]
else:
    remote_addr = ips[-2]

...
        'REMOTE_ADDR':
            remote_addr

Another option is to use event["requestContext"]["identity"]["sourceIp"], although I've not tested whether that is spoofable or not.

I do not know if this is already implemented but I have not found it in the docs. My apologies in advance If it can be done easier.

I find myself including a serverless env variable reading an opts argument to check if my code is running locally to disable some functionalities (logs, remote module loading...)

environment:
  DEBUG_SERVER: ${opt:debugServer, 'false'}

It would be nice if this were built in transparently from serverless via (werkzeug or the wsgi handler), I have checked the environment of the lambda function and I haven't seen anything like this.

I can help with this if you agree and confirm the proper design.

PS: Thanks for your amazing work, you saved us a lot of time sir.

Currently, the "sls wsgi serve" command has an error when calling wsgi.

It will return: "Unable to import werkzeug (run: pip install werkzeug)"
even though the werkzeug requirement is in the requirements.txt file of the wsgi package.

The steps to directly reproduce on a fresh Ubuntu 16.04 VirtualBox Install can be found here:
https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/

Just for kicks:
-I installed werkzeug locally and sudo cp'd it into the serverless-wsgi main directory and there was another requirement that needed to be installed (boto3, botocore, click, etc). I did this for a couple of iterations before coming here.
-I pip installed everything needed for the ^^ dependency tree failure in a virtualenv and pip freezed the venv into the serverless-wsgi requirements.txt , also no avail.

Due to a quirk in API Gateway, multiple set-cookie's will be dropped to a single one. Mangling the case of the set-cookie header is sufficient to pass them through though.

This pluging should do the case mangling on the way out, perhaps via middleware as Zappa does, to remedy this. Here's what my PR to Zappa to do this looked like:
Miserlou/Zappa#636

The final code is fairly simple, perhaps it can be more directly incorporated into the WSGI handler:
https://github.com/Miserlou/Zappa/blob/master/zappa/middleware.py

Hi.

I see the following error in my Lambda function Cloudwatch Logs when using this serverless plugin.

START RequestId: f5509b02-3c75-11e7-8d94-d97b07264760 Version: $LATEST
Syntax error in module 'wsgi': Missing parentheses in call to 'print' (wsgi.py, line 118)

END RequestId: f5509b02-3c75-11e7-8d94-d97b07264760

It seems that the print statement without parenthesis is not compatible with Python 3.6.

Any chance you'll be porting the plugin to a 3.6 compatible version soon?

I was trying the new django management commands.

My serverless.yml contains environmental variables for different stages and in order to run the wsgi server locally I do: sls wsgi serve -s local.

I would also like to execute the django management commands locally such as with sls wsgi manage -c "check --list-tags" -s local.

It seems to not work out as I get the following error:

Function not found: arn:aws:lambda:eu-central-1:xxxxxx:function:cookbook-local-app

Is this just not possible or am I missing something?

I can only get this to work on Centos 7. On any other OS it never creates the .requirements folder.

#As Root
yum install nodejs
yum install npm
npm install -g npm@lts
npm install -g serverless

#Local
mkdir web
cd web/
virtualenv web
source web/bin/activate
pip install flask
npm install --save serverless-wsgi
pip install werkzeug
pip install virtualenv
serverless create --template aws-python
vi serverless.yml
vi requirements.txt (Flask==0.11.1)
vi api.py (same code as readme)
serverless deploy

Using Flask, I cannot see a way to access the raw 'event' from within a function, although the 'context' is available:

        'context':
            context,

So for example, I can't get to low-level things like event["requestContext"]["identity"]["sourceIp"]

Have I overlooked something? Otherwise, could it be added like this:

        'event':
            event,

Has file uploads / binary data been tested with this? Last I've tried, I was unable to get binary data through api gateway.

Everything is working on CentOS 7.x. If I run the exact same sample code on an Amazon Linux instance the .requirements folder is missing from the zip file. I ran the python requirements.py requirements.txt .requirements command and it just creates and empty .requirements folder.

One thing that had to be done was to run yum remove python27-virtualenv. This fixed issues with the serverless framework. I believe the issue is related to some pip package issue. I have listed the pre-installed pip packeages on the system below.

aws-cfn-bootstrap (1.4)
awscli (1.11.83)
Babel (0.9.4)
backports.ssl-match-hostname (3.4.0.2)
boto (2.42.0)
botocore (1.5.46)
chardet (2.0.1)
cloud-init (0.7.6)
colorama (0.2.5)
configobj (4.7.2)
docutils (0.11)
ecdsa (0.11)
futures (3.0.3)
gyp (0.1)
iniparse (0.3.1)
jmespath (0.9.2)
jsonpatch (1.2)
jsonpointer (1.0)
kitchen (1.1.1)
lockfile (0.8)
paramiko (1.15.1)
PIL (1.1.6)
pip (9.0.1)
ply (3.4)
pyasn1 (0.1.7)
pycrypto (2.6.1)
pycurl (7.19.0)
pygpgme (0.3)
pyliblzma (0.5.3)
pystache (0.5.3)
python-daemon (1.5.2)
python-dateutil (2.1)
pyxattr (0.5.0)
PyYAML (3.10)
requests (1.2.3)
rsa (3.4.1)
setuptools (12.2)
simplejson (3.6.5)
six (1.8.0)
urlgrabber (3.10)
urllib3 (1.8.2)
virtualenv (15.1.0)
yum-metadata-parser (1.1.4)

Spent a good few hours trying to work out why an image/svg+xml MIME was returning garbage:

Should return something like:
<?xml version="1.0"?><svg width="1125" height="432" xmlns=...

But was returning:
PD94bWwgdmVyc2lvbj0iMS4wIj8 ....

I then read the following in the docs:

Text MIME types
By default, all MIME types starting with text/ and the following whitelist are sent through API Gateway in plain text. All other MIME types will have their response body base64 encoded (and the isBase64Encoded API Gateway flag set) in order to be delivered by API Gateway as binary data.

It then dawned on me that image/svg+xml needs to be returned as text, not binary (like most images).

Fixed by adding the following to serverless.yml:

    textMimeTypes:
    - image/svg+xml

Whilst this may be an edge case, it may be helpful to add image/svg+xml to the whitelist as I went round and round in circles in my attempt to solve this.

Since SVG images are mainly used in use cases where you need to real-time build an image in code (instead of serving one directly from s3) its seems like a good fit for the whitelist in my opinion.

Hi I have a flask app that returns content type application/zip but when i run it using serverless-wsgi it seems like the content type is changed to text/plain. any idea what i might be doing wrong? Thanks

Add command to automatically determine routes from (Flask) app and update serverless.yml.

Or... is there difference between the two?

Hi @logandk, community members,

Thanks for creating and open sourcing this amazing package :)

While working with open whisk, I had this little issue :

$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Packaging service...
Serverless: Compiling Functions...
 
  Serverless Error ---------------------------------------
 
  Function handler (wsgi.py) does not exist.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.2.2
     Serverless Version:     1.15.1

It said that wsgi.py does not exist, but I verified that the file was present in the root directory when I ran the command.

Can you please help with this ?

package has requirement - werkzeug, but this is not written
in setup.py

After testing the plugin with the project very much similar to the one in example, in which local tests work correctly, after deployment to AWS I get the following message on call:

{
  "errorMessage": "Unable to import module 'wsgi'"
}

Detailed log:

START RequestId: 6b36272e-920d-11e6-9aed-0d5b9d25097b Version: $LATEST
Unable to import module 'wsgi': No module named datastructures

END RequestId: 6b36272e-920d-11e6-9aed-0d5b9d25097b
REPORT RequestId: 6b36272e-920d-11e6-9aed-0d5b9d25097b  Duration: 0.41 ms   Billed Duration: 100 ms     Memory Size: 1024 MB    Max Memory Used: 15 MB  

It seems that the requirements are not correctly visible to WSGI. Everything in the deployment package seems to be in the right place.

Thanks for the help.

When I use this include/exclude configuration:

package:
  exclude:
    - '**' # everything
  include:
    - '*.py'

....requirements and .wsgi_app aren't packaged. I have to use this:

package:
  exclude:
    - '**' # everything
  include:
    - '*.py'
    - '.requirements/**' # Serverless WSGI
    - '.wsgi_app' # Serverless WSGI

It seems useful to include .requirements/** and .wsgi_app, even when that's not specified. In the same vein, it seems useful to exclude node_modules/serverless-wsgi/**. (Similarly, Serverless itself excludes .serverless\**, even when that's not specified.

Suddenly getting this error again on v1.6.1

Rolling back to 1.6.0 fixes the issue

I started having this issue today when deploying python lambda functions, and it's hard to say what caused this, since I didn't update any relevant configuration. The cloudwatch logs also only show:

Unable to import module 'wsgi': No module named 'werkzeug'

with no other backtrace, and this seems to happen before my application code is evaluated. I also tried to revert to the latest stable release, and it also doesn't work.

Did something in AWS change which caused this error?