clericpy / zipapps Goto Github PK

Package your python code into an executable zip file (with the requirements). Based on pure python zipapp, for Python3.6+.

License: MIT License

Python 100.00%

python python3 zipapp pep-441 standalone package venv pyz distribution

zipapps's People

Contributors

Stargazers

Watchers

Forkers

pombredanne tomtomzhang1970 shalevy1 alan0526 igoforth

zipapps's Issues

Support --dump-config and --load-config

Support clear self `pyz` after running

Thanks!

Just want to say that this has recently made some projects of mine MUCH easier in terms of distribution.
Thanks so much!

refactor environment variables template and interval variables(with string.Template)

Custom cache folder name for different caller, such as $pid, $cwd

And support ENV_ARGS to rewrite it.(enhance ZIPAPPS_CACHE )

Try freezing requirements with zipapps

zipapps with nuitka

build with nuitka, and the requirements from Zipapps

Permission error while using `lazy-install` between different users

Can zippos solve these 2 issues

I have two issues with shiv
linkedin/shiv#209
linkedin/shiv#208
Does zipapps support these functionalities?

ImportError: No module named 'pip._internal' while using pip<=19

Can not set .pyw module as entrypoint

Support event hook at the start / end stage

And use it to refactor old features like --clear-zipapps-cache

Issues with psycopg2-binary: libpq not found

# python3 -m zipapps -c -a conf-template.yml -a manager.py -o manager.pyz -m manager:app -u AUTO -r requirements.txt 
[...]
[INFO]: these names will be unzipped while running: yaml,psycopg2,_ldap.cpython-38-x86_64-linux-gnu.so
[INFO] Temp cache has been cleaned. (<TemporaryDirectory '/tmp/tmpflsdglju'>)
========== Successfully built `manager.pyz` ==========
# python3 manager.pyz
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "manager.pyz/__main__.py", line 94, in <module>
  File "manager.pyz/__main__.py", line 62, in main
  File "<frozen zipimport>", line 259, in load_module
  File "/app/manager.pyz/manager.py", line 7, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/app/manager.pyz/ldap2pg/__init__.py", line 1, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/app/manager.pyz/ldap2pg/config.py", line 22, in <module>
  File "/app/zipapps_cache/manager/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: libpq-6f24e430.so.5.13: cannot open shared object file: No such file or directory

unable to run zippapps with pywin32

@ClericPy - I struggle to overcome pywin32 issue when using zipapps.

When I installed pywin32 module and package it by runing below command
python -m zipapps -o demo.pyz -m "src.main:main" -d -c -u=* -r .\requirements.txt -a "src"

then running python demo.pyz outside virtual environment or on different machine will produce below error

Traceback (most recent call last):
  File "C:\Users\pythonuser\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\pythonuser\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File ".\demo.pyz\__main__.py", line 101, in <module>
    main()
  File ".\demo.pyz\__main__.py", line 66, in main
    import src.main
  File "F:\myapps\zipapps_cache\demo\src\main.py", line 6, in <module>
    import win32com.client as win32
  File "F:\myapps\zipapps_cache\demo\_zipapps_lazy_pip\3.8_Windows\win32com\__init__.py", line 5, in <module>
    import win32api, sys, os
ModuleNotFoundError: No module named 'win32api'

Do you have any work around this or any suggestions?

Does zipapps need `--lock`?

https://stackoverflow.com/questions/32390291/pip-freeze-for-only-project-requirements

Not necessary.

certifi lib can not create cert file in zip file and create them to /tmp/tmp****

unzip that lib, or it will create a new temp file while each running

add --unzip-exclude

unzip except given names, the opposite of --unzip

-ux=source_code -u=*

Crash when using importlib.metadata.version

When using

from importlib.metadata import version
version("name_of_my_package")

I get a crash importlib.metadata.PackageNotFoundError: No package metadata was found for name_of_my_package

I first create the wheel of my package and then create the

   zipapps --output dist/$(PACKAGE_NAME).pyz -u=AUTO
        -p '/usr/bin/env python3'
        -m name_of_my_package.cli:cli 
        -r dist/prod-requirements.txt 
        dist/name_of_my_package-*.whl

I think name_of_my_package.whl is well installed I see it in the pip install output, but it seems like the importlib does not find it when executed in the pyz generated by zipapps. It works with shiv.

pip install failed but build success?

auto-clear to rm unzip cache after running

fail to try this, for pyd/so file can not be removed while using

Auto clear cache fold after running

maybe two ways:

use a memory file system, tmpfs
auto delete folders after runpy.

Does zipapps need `--clear`?

Not very useful.

rm app.pyz ./zipapps_cache

py to pyc

Could you compile py to pyc and then zip pyc to pyz?

pkg_resources.DistributionNotFound: The 'ldap2pg' distribution was not found and is required by the application

It seems that ldap2pg has not been included:

# grep -r ldap2pg requirements.txt 
ldap2pg==5.6 \
# python3 -m zipapps -c -a conf-template.yml -a manager.py -o manager.pyz -m manager:app -u '*' -r requirements.txt 
========== Start building `manager.pyz` with zipapps version <2021.04.29> ==========
[INFO]: the arg `unzip_path` has been changed to `zipapps_cache` by default.
[INFO]: output path is `manager.pyz`, you can reset it with the arg `output`.
Collecting click==8.0.1
  Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting ldap2pg==5.6
  Using cached ldap2pg-5.6-py2.py3-none-any.whl (45 kB)
Collecting psycopg2-binary==2.9.1
  Using cached psycopg2_binary-2.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
Collecting pyasn1==0.4.8
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules==0.2.8
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting python-ldap==3.3.1
  Using cached python-ldap-3.3.1.tar.gz (379 kB)
Collecting pyyaml==5.4.1
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting typer==0.4.0
  Using cached typer-0.4.0-py3-none-any.whl (27 kB)
Building wheels for collected packages: python-ldap
  Building wheel for python-ldap (setup.py) ... done
  Created wheel for python-ldap: filename=python_ldap-3.3.1-cp38-cp38-linux_x86_64.whl size=342147 sha256=47ad3a2f7238615aedfda76e8b1987a97f4747790ee8ec43f954ec88324d0d04
  Stored in directory: /root/.cache/pip/wheels/db/e9/83/84ab9509d4cba1c1f74e929ffa8890184c23b95cbcc5d0b0a9
Successfully built python-ldap
Installing collected packages: pyasn1, pyasn1-modules, pyyaml, python-ldap, click, typer, psycopg2-binary, ldap2pg
Successfully installed click-8.0.1 ldap2pg-5.6 psycopg2-binary-2.9.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 python-ldap-3.3.1 pyyaml-5.4.1 typer-0.4.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[WARN]: .pyd/.so files may be imported incorrectly, set `--unzip=psycopg2,_ldap.cpython-38-x86_64-linux-gnu.so,yaml` or `--unzip=AUTO` to fix it. {'psycopg2': {'.so': 1}, '_ldap.cpython-38-x86_64-linux-gnu.so': {'.so': 1}, 'yaml': {'.so': 1}}
[INFO]: these names will be unzipped while running: *
[INFO] Temp cache has been cleaned. (<TemporaryDirectory '/tmp/tmpzzg5v6jn'>)
========== Successfully built `manager.pyz` ==========
[root@e18ad3fcfb4e app]# python3 manager.pyz 
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "manager.pyz/__main__.py", line 94, in <module>
    main()
  File "manager.pyz/__main__.py", line 62, in main
    import manager
  File "/app/zipapps_cache/manager/manager.py", line 7, in <module>
    import ldap2pg
  File "/app/zipapps_cache/manager/ldap2pg/__init__.py", line 1, in <module>
    from .config import __version__, __dist__
  File "/app/zipapps_cache/manager/ldap2pg/config.py", line 40, in <module>
    __dist__ = get_distribution('ldap2pg')
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 481, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 357, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ldap2pg' distribution was not found and is required by the application

attampt to run zip file in memory with curl and xargs

something like curl -s http://s3-bucket/app.pyz| python, sometimes use it to avoid leaking source code in unsafe environment.

simplify the readme

tooooooooooooo long to read

search for needed package files locally instead of using pip every time

Requirements of a package can be searched with importlib.metadata.requires and files can be searched with importlib.metadata.files, so it is possible to gather needed files without using pip to install into a virtual environment.

Here is a script in my project that gathers needed files into cache directory and uses zipapps to pack. Just to illustate how to search package files with the way mentioned above. This script works on Python 3.11, earlier versions may use backported importlib-metadata instead.

import os, sys, platform
from shutil import rmtree
from importlib.util import find_spec
from importlib.metadata import files as pkg_files, requires as pkg_requires
from packaging.requirements import Requirement
from pathlib import Path

import yaml
from zipapps import ZipApp

HERE = Path(__file__).parent
PROJ_ROOT = HERE.parent
sys.path.append(str(PROJ_ROOT))
PROJ_DIST = PROJ_ROOT / 'dist'
PYVER = sys.version_info

def module_exists(modname: str):
    try: return find_spec(modname) is not None
    except: return False

def slash_to_dot(modname: str):
    return modname.replace('/', '.').replace('\\', '.')

def top_name(modname: str):
    return modname.split('.', maxsplit=1)[0]

def get_module_path(modname: str):
    spec = find_spec(modname)
    if spec is None: raise ValueError
    if spec.origin is None: raise ValueError
    file_path = Path(spec.origin)
    if file_path.name == '__init__.py':
        return file_path.parent
    return file_path

def read_yaml(pth: Path):
    return yaml.load(pth.read_text(encoding='utf-8'), yaml.CLoader)

def check_req(req: Requirement, extras: set[str]):
    return not req.marker or req.marker.evaluate({'extra': ''}) \
        or any(req.marker.evaluate({'extra': extra}) for extra in extras)

def search_direct_deps(pkg: Requirement) -> list[Requirement]:
    try: reqexprs = pkg_requires(pkg.name)
    except: return []
    if reqexprs is None: return []
    return [req for req in map(Requirement, reqexprs) if check_req(req, pkg.extras)]

def search_all_deps(pkgs: list[Requirement]):
    result = dict[str, set[str]]()
    stack = [iter(pkgs)]
    while stack:
        meet = next(stack[-1], None)
        if meet is None:
            stack.pop(); continue
        met_extras = result.get(meet.name, None)
        if met_extras is None:
            result[meet.name] = meet.extras.copy()
            stack.append(iter(search_direct_deps(meet)))
        else:
            meet.extras -= met_extras
            if meet.extras:
                met_extras.update(meet.extras)
                stack.append(iter(search_direct_deps(meet)))
    return result

def copy_package(pkgname: str, dest: Path):
    pkgfiles = pkg_files(pkgname)
    if pkgfiles is None: return
    dest_abs = dest.resolve()
    for pkgfile in pkgfiles:
        file_abs = (dest_abs / pkgfile).resolve()
        if not file_abs.is_relative_to(dest_abs): continue
        file_abs.parent.mkdir(parents=True, exist_ok=True)
        file_abs.write_bytes(pkgfile.read_binary())


PROJ_DIST.mkdir(exist_ok=True)
config = read_yaml(HERE / f'{sys.argv[1]}.yaml')
main_name: str = config['main']
file_name = f'{main_name}_py{PYVER.major}{PYVER.minor}'
modules: list[str] = [main_name, *config['modules']]
print(f'递归查找包的依赖...')
packages = search_all_deps([Requirement(pkg.lower()) for pkg in config['site-packages']])
print(f'已找到依赖的包：{packages}')
module_paths = [get_module_path(name) for name in modules]
cache_path = PROJ_DIST / f'{file_name}.cache'
if cache_path.exists(): rmtree(cache_path)
cache_path.mkdir(exist_ok=True)
print('复制依赖的包到缓存目录...')
for pkg in packages: copy_package(pkg, cache_path)

ZipApp(
    compressed=True,
    cache_path=str(cache_path),
    includes=','.join(str(p) for p in module_paths),
    main=f'{main_name}.__main__',
    output=str(PROJ_DIST / f'{file_name}.pyz'),
    unzip='AUTO',
).build()

An example config file for this script:

main: mymod
modules:
  - local_module
site-packages:
  - ipython
  - pyyaml
  - pillow
  - prompt-toolkit
  - pycryptodome
  - requests
  - zipp