Clash on `__pycache__` directory about serverless-wsgi HOT 8 CLOSED

Yes, perhaps we should keep a blacklist of what to omit when packaging requirements. Regarding .pyc files, actually Zappa (at one point, at least) would compile the bytecode and only package the .pyc files to reduce package size and improve performance. The bytecode files are cross-platform compatible, so they shouldn't be rebuilt on the server-side AFAIK.

So ignoring .pyc files should probably stay opt-in through an exclude rule, but let's avoid these packaging conflicts by default.

from serverless-wsgi.

perhaps we should keep a blacklist of what to omit when packaging requirements

Yes, although in this case I had explicitly defined only include: requirements:

package:
  exclude:
    - "**"
  include:
    - "*.py"

I was surprised when the presence of __pycache__ caused a conflict error from serverless-wsgi, when __pycache__ was not going to be bundled anyway.

Now: python3.2 changed from having .pyc files adjacent to the .py files, to having a __pycache__ directory.

I wondered why a requirements bundle would ever have a top-level __pycache__ directory. The answer is because I'm using xmltodict, and this plops xmltodict.py at the top level, rather than using a package (xmltodict/__init__.py):

$ unzip -v .serverless/xxxxx.zip  | grep " __pycache__"
   13913  Defl:N     6989  50% 1980-01-01 00:00 81ca3cca  __pycache__/xmltodict.cpython-35.pyc

If you needed to upload your own __pycache__ directory, and also the requirements has a top-level __pycache__ directory, that's a pain. serverless would have to merge the contents of the two directories.

The bytecode files are cross-platform compatible, so they shouldn't be rebuilt on the server-side AFAIK

That's good. Is that true even across python versions - in particular if I create .pyc files on python 3.5 under Ubuntu 16.04, they are usable on python 3.6 under Lambda? If so then I don't really mind that they're there.

Omitting them would make the zip file a tiny bit smaller, which might help if you're getting near the 50MB limit, but is a minor optimisation. Adding a few milliseconds to the first time a lamdba starts, I don't really care about.

from serverless-wsgi.

Unfortunately the fix doesn't work for me.

$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Linking required Python packages...

  Serverless Error ---------------------------------------

  Unable to link dependency '__pycache__' because a file by the same name exists in this service

  Get Support --------------------------------------------
... etc

$ npm -g list | grep wsgi
└─┬ [email protected]

$ grep pycache /usr/lib/node_modules/serverless-wsgi/requirements.py
        'setuptools-*', 'easy_install.*', '__pycache__']

Workaround is still: rm -rf __pycache__/ at the top level of the project before sls deploy

from serverless-wsgi.

@candlerb Could you try removing .requirements and __pycache__ and try again?

from serverless-wsgi.

That's the point: if I rm -rf __pycache__ then it works fine. If I don't, I get the failure.

from serverless-wsgi.

Yes, but if __pycache__ was already in .requirements from the previous version of serverless-wsgi, it will still fail. Now, __pycache__ should not end up in .requirements in the first place. I.e. try removing .requirements and package. On subsequent runs, it should not fail like before.

from serverless-wsgi.

On second thought, since __pycache__ may be created again after building the .requirements folder, a blacklist in the plugin (where the symlinks are created) is still needed. The current blacklist only skips the __pycache__ folder that pip creates initially.

from serverless-wsgi.

Ah I see.

Initial status: both .requirements and __pycache__ exist.

$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Linking required Python packages...

  Serverless Error ---------------------------------------

  Unable to link dependency '__pycache__' because a file by the same name exists in this service

Now I mv .requirements .requirements.old

$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Linking required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Unlinking required Python packages...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (11.15 MB)...

This works, thanks! (The top-level __pycache__ directory still exists)

I notice that the .requirements.old directory contains a __pycache__ directory, but the new .requirements does not.

from serverless-wsgi.