Non-WSGI code does not work after `sls function deploy -f name` about serverless-wsgi HOT 11 CLOSED

Note: redeploying just the WSGI function is fine.

$ sls deploy function -f web
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Packaging function: web...
Serverless: Excluding development dependencies...
Serverless: Uploading function: web (2.06 MB)...
Serverless: Successfully deployed function: web

from serverless-wsgi.

Indeed, we should be packaging the requirements for the non-WSGI function here as well

from serverless-wsgi.

Thanks.

BTW I had a quick look how serverless-python-requirements avoids the need to put .requirements into sys.path. AFAICS, it seems to be simply that it puts all the requirements at the top level:

$ unzip -v .serverless/my-sql.zip
Archive:  .serverless/my-sql.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      84  Defl:N       77   8% 1980-01-01 00:00 0163f8c9  env_dev.yml
     639  Defl:N      418  35% 1980-01-01 00:00 22eb0fcd  env_dev.yml~
     544  Defl:N      315  42% 1980-01-01 00:00 51ec2ddf  handler.py
      10  Defl:N       12 -20% 1980-01-01 00:00 d27f2d5e  PyMySQL-0.7.11.dist-info/DESCRIPTION.rst
       4  Defl:N        6 -50% 1980-01-01 00:00 c2971fc7  PyMySQL-0.7.11.dist-info/INSTALLER
     534  Defl:N      311  42% 1980-01-01 00:00 37a15918  handler.py~
     850  Defl:N      330  61% 1980-01-01 00:00 c27f473b  PyMySQL-0.7.11.dist-info/METADATA
    6278  Defl:N     2522  60% 1980-01-01 00:00 8c0e447f  PyMySQL-0.7.11.dist-info/RECORD
     935  Defl:N      429  54% 1980-01-01 00:00 a29c8b8a  PyMySQL-0.7.11.dist-info/metadata.json
       8  Defl:N       10 -25% 1980-01-01 00:00 f50e859b  PyMySQL-0.7.11.dist-info/top_level.txt
     110  Defl:N       95  14% 1980-01-01 00:00 1382dbab  PyMySQL-0.7.11.dist-info/WHEEL
    4516  Defl:N     1953  57% 1980-01-01 00:00 07a9d10f  pymysql/__init__.py
    5033  Defl:N     2810  44% 1980-01-01 00:00 fb588bea  pymysql/__pycache__/__init__.cpython-35.pyc
     668  Defl:N      436  35% 1980-01-01 00:00 01f3dc51  pymysql/__pycache__/_compat.cpython-35.pyc
    4440  Defl:N     2116  52% 1980-01-01 00:00 ab9efe58  pymysql/__pycache__/_socketio.cpytho
...

from serverless-wsgi.

Yeah, I did consider doing that originally, however I went with a separate directory to avoid accidentally overwriting files in the project with any of the files from the requirements and to keep the deployment package organized. It would be nice to avoid having to add .requirements to sys.path, but I'm not convinced yet that just dumping everything in the root directory is a great approach.

from serverless-wsgi.

Could it be done by setting PYTHONPATH environment variable?

(Even if that has to be explicitly set in serverless.yml, that's still probably better than changing the code)

from serverless-wsgi.

Thanks for the suggestion. Since PYTHONPATH needs to be set before the Python interpreter is invoked, and Lambda is probably not invoking the interpreter on each request (in order to keep start-up times low), it may not work - but it's worth trying out.

Alternatively, we can extract the part of wsgi.py that sets sys.path into its own script, say wsgi_requirements.py which is bundled along with wsgi.py, and used by wsgi.py as well. For non-wsgi scripts, using the packaged requirements would simply be an import wsgi_requirements statement. It's still not completely transparent, but simpler.

I'll also reconsider the serverless-python-requirements approach of putting requirements in the package root, perhaps simply throwing an error if there are file name conflicts will suffice to avoid any unexpected behaviour.

from serverless-wsgi.

PYTHONPATH=.requirements doesn't work here, but you can see why if I change my api.py:

def handler(event, context):
    import os, sys
    return {
      "PYTHONPATH":os.environ['PYTHONPATH'],
      "sys.path":sys.path,
    }

You get:

$ sls invoke -f api
{
    "PYTHONPATH": ".requirements",
    "sys.path": [
        "/var/task",
        "/var/runtime/awslambda",
        "/.requirements",
        "/var/lang/lib/python36.zip",
        "/var/lang/lib/python3.6",
        "/var/lang/lib/python3.6/lib-dynload",
        "/var/lang/lib/python3.6/site-packages"
    ]
}

I don't know why / is prepended, but it works properly if I set PYTHONPATH to /var/task/.requirements. Maybe that's a reasonable solution.

from serverless-wsgi.

That's great, I was surprised to see that it did have an effect. The downside is that we'll need to hardcode a path that is implementation-specific to AWS lambda, it may (unlikely) change and will most likely be different on other providers. I think that putting packages in the root and checking that no files are overwritten may be the most reliable long-term solution.

from serverless-wsgi.

The downside is that we'll need to hardcode a path that is implementation-specific to AWS lambda

True, or you could just document that the user should set PYTHONPATH=/var/task/.requirements (at least that's better than the current 4-line code change).

I think that putting packages in the root and checking that no files are overwritten may be the most reliable long-term solution

On balance, I think you're right. Speaking as a user, it's helpful if this project and serverless-python-requirements can work in similar ways.

from serverless-wsgi.

I found a problem with setting PYTHONPATH=/var/task/.requirements globally in a mixed WSGI/non-WSGI app, and my WSGI app tries to use the AWS-provided instance of boto3, it fails to load:

Unable to import module 'wsgi': No module named 'boto3'

It turns out that AWS sets PYTHONPATH=/var/runtime by default. Setting PYTHONPATH=/var/runtime:/var/task/.requirements solves this.

(But this is another reason not to mess with PYTHONPATH)

from serverless-wsgi.

Thanks for the feedback, I'm wondering how many of the other providers except for AWS that allow settings ENV vars and whether they'll also set them before starting the interpreter

from serverless-wsgi.