clericpy / zipapps Goto Github PK
View Code? Open in Web Editor NEWPackage your python code into an executable zip file (with the requirements). Based on pure python zipapp, for Python3.6+.
License: MIT License
Package your python code into an executable zip file (with the requirements). Based on pure python zipapp, for Python3.6+.
License: MIT License
Just want to say that this has recently made some projects of mine MUCH easier in terms of distribution.
Thanks so much!
Custom cache folder name for different caller, such as $pid, $cwd
And support ENV_ARGS to rewrite it.(enhance ZIPAPPS_CACHE
)
build with nuitka, and the requirements from Zipapps
I have two issues with shiv
linkedin/shiv#209
linkedin/shiv#208
Does zipapps support these functionalities?
And use it to refactor old features like --clear-zipapps-cache
# python3 -m zipapps -c -a conf-template.yml -a manager.py -o manager.pyz -m manager:app -u AUTO -r requirements.txt
[...]
[INFO]: these names will be unzipped while running: yaml,psycopg2,_ldap.cpython-38-x86_64-linux-gnu.so
[INFO] Temp cache has been cleaned. (<TemporaryDirectory '/tmp/tmpflsdglju'>)
========== Successfully built `manager.pyz` ==========
# python3 manager.pyz
Traceback (most recent call last):
File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "manager.pyz/__main__.py", line 94, in <module>
File "manager.pyz/__main__.py", line 62, in main
File "<frozen zipimport>", line 259, in load_module
File "/app/manager.pyz/manager.py", line 7, in <module>
File "<frozen zipimport>", line 259, in load_module
File "/app/manager.pyz/ldap2pg/__init__.py", line 1, in <module>
File "<frozen zipimport>", line 259, in load_module
File "/app/manager.pyz/ldap2pg/config.py", line 22, in <module>
File "/app/zipapps_cache/manager/psycopg2/__init__.py", line 51, in <module>
from psycopg2._psycopg import ( # noqa
ImportError: libpq-6f24e430.so.5.13: cannot open shared object file: No such file or directory
@ClericPy - I struggle to overcome pywin32 issue when using zipapps.
When I installed pywin32 module and package it by runing below command
python -m zipapps -o demo.pyz -m "src.main:main" -d -c -u=* -r .\requirements.txt -a "src"
then running python demo.pyz
outside virtual environment or on different machine will produce below error
Traceback (most recent call last):
File "C:\Users\pythonuser\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\pythonuser\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File ".\demo.pyz\__main__.py", line 101, in <module>
main()
File ".\demo.pyz\__main__.py", line 66, in main
import src.main
File "F:\myapps\zipapps_cache\demo\src\main.py", line 6, in <module>
import win32com.client as win32
File "F:\myapps\zipapps_cache\demo\_zipapps_lazy_pip\3.8_Windows\win32com\__init__.py", line 5, in <module>
import win32api, sys, os
ModuleNotFoundError: No module named 'win32api'
Do you have any work around this or any suggestions?
unzip that lib, or it will create a new temp file while each running
unzip except given names, the opposite of --unzip
-ux=source_code -u=*
Hi
When using
from importlib.metadata import version
version("name_of_my_package")
I get a crash importlib.metadata.PackageNotFoundError: No package metadata was found for name_of_my_package
I first create the wheel of my package and then create the
zipapps --output dist/$(PACKAGE_NAME).pyz -u=AUTO
-p '/usr/bin/env python3'
-m name_of_my_package.cli:cli
-r dist/prod-requirements.txt
dist/name_of_my_package-*.whl
I think name_of_my_package.whl is well installed I see it in the pip install output, but it seems like the importlib does not find it when executed in the pyz generated by zipapps. It works with shiv.
fail to try this, for pyd/so file can not be removed while using
maybe two ways:
Not very useful.
rm app.pyz ./zipapps_cache
Could you compile py to pyc and then zip pyc to pyz?
It seems that ldap2pg has not been included:
# grep -r ldap2pg requirements.txt
ldap2pg==5.6 \
# python3 -m zipapps -c -a conf-template.yml -a manager.py -o manager.pyz -m manager:app -u '*' -r requirements.txt
========== Start building `manager.pyz` with zipapps version <2021.04.29> ==========
[INFO]: the arg `unzip_path` has been changed to `zipapps_cache` by default.
[INFO]: output path is `manager.pyz`, you can reset it with the arg `output`.
Collecting click==8.0.1
Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting ldap2pg==5.6
Using cached ldap2pg-5.6-py2.py3-none-any.whl (45 kB)
Collecting psycopg2-binary==2.9.1
Using cached psycopg2_binary-2.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
Collecting pyasn1==0.4.8
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules==0.2.8
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting python-ldap==3.3.1
Using cached python-ldap-3.3.1.tar.gz (379 kB)
Collecting pyyaml==5.4.1
Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting typer==0.4.0
Using cached typer-0.4.0-py3-none-any.whl (27 kB)
Building wheels for collected packages: python-ldap
Building wheel for python-ldap (setup.py) ... done
Created wheel for python-ldap: filename=python_ldap-3.3.1-cp38-cp38-linux_x86_64.whl size=342147 sha256=47ad3a2f7238615aedfda76e8b1987a97f4747790ee8ec43f954ec88324d0d04
Stored in directory: /root/.cache/pip/wheels/db/e9/83/84ab9509d4cba1c1f74e929ffa8890184c23b95cbcc5d0b0a9
Successfully built python-ldap
Installing collected packages: pyasn1, pyasn1-modules, pyyaml, python-ldap, click, typer, psycopg2-binary, ldap2pg
Successfully installed click-8.0.1 ldap2pg-5.6 psycopg2-binary-2.9.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 python-ldap-3.3.1 pyyaml-5.4.1 typer-0.4.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[WARN]: .pyd/.so files may be imported incorrectly, set `--unzip=psycopg2,_ldap.cpython-38-x86_64-linux-gnu.so,yaml` or `--unzip=AUTO` to fix it. {'psycopg2': {'.so': 1}, '_ldap.cpython-38-x86_64-linux-gnu.so': {'.so': 1}, 'yaml': {'.so': 1}}
[INFO]: these names will be unzipped while running: *
[INFO] Temp cache has been cleaned. (<TemporaryDirectory '/tmp/tmpzzg5v6jn'>)
========== Successfully built `manager.pyz` ==========
[root@e18ad3fcfb4e app]# python3 manager.pyz
Traceback (most recent call last):
File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "manager.pyz/__main__.py", line 94, in <module>
main()
File "manager.pyz/__main__.py", line 62, in main
import manager
File "/app/zipapps_cache/manager/manager.py", line 7, in <module>
import ldap2pg
File "/app/zipapps_cache/manager/ldap2pg/__init__.py", line 1, in <module>
from .config import __version__, __dist__
File "/app/zipapps_cache/manager/ldap2pg/config.py", line 40, in <module>
__dist__ = get_distribution('ldap2pg')
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 481, in get_distribution
dist = get_provider(dist)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 357, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ldap2pg' distribution was not found and is required by the application
something like curl -s http://s3-bucket/app.pyz| python
, sometimes use it to avoid leaking source code in unsafe environment.
tooooooooooooo long to read
Requirements of a package can be searched with importlib.metadata.requires
and files can be searched with importlib.metadata.files
, so it is possible to gather needed files without using pip to install into a virtual environment.
Here is a script in my project that gathers needed files into cache directory and uses zipapps
to pack. Just to illustate how to search package files with the way mentioned above. This script works on Python 3.11, earlier versions may use backported importlib-metadata
instead.
import os, sys, platform
from shutil import rmtree
from importlib.util import find_spec
from importlib.metadata import files as pkg_files, requires as pkg_requires
from packaging.requirements import Requirement
from pathlib import Path
import yaml
from zipapps import ZipApp
HERE = Path(__file__).parent
PROJ_ROOT = HERE.parent
sys.path.append(str(PROJ_ROOT))
PROJ_DIST = PROJ_ROOT / 'dist'
PYVER = sys.version_info
def module_exists(modname: str):
try: return find_spec(modname) is not None
except: return False
def slash_to_dot(modname: str):
return modname.replace('/', '.').replace('\\', '.')
def top_name(modname: str):
return modname.split('.', maxsplit=1)[0]
def get_module_path(modname: str):
spec = find_spec(modname)
if spec is None: raise ValueError
if spec.origin is None: raise ValueError
file_path = Path(spec.origin)
if file_path.name == '__init__.py':
return file_path.parent
return file_path
def read_yaml(pth: Path):
return yaml.load(pth.read_text(encoding='utf-8'), yaml.CLoader)
def check_req(req: Requirement, extras: set[str]):
return not req.marker or req.marker.evaluate({'extra': ''}) \
or any(req.marker.evaluate({'extra': extra}) for extra in extras)
def search_direct_deps(pkg: Requirement) -> list[Requirement]:
try: reqexprs = pkg_requires(pkg.name)
except: return []
if reqexprs is None: return []
return [req for req in map(Requirement, reqexprs) if check_req(req, pkg.extras)]
def search_all_deps(pkgs: list[Requirement]):
result = dict[str, set[str]]()
stack = [iter(pkgs)]
while stack:
meet = next(stack[-1], None)
if meet is None:
stack.pop(); continue
met_extras = result.get(meet.name, None)
if met_extras is None:
result[meet.name] = meet.extras.copy()
stack.append(iter(search_direct_deps(meet)))
else:
meet.extras -= met_extras
if meet.extras:
met_extras.update(meet.extras)
stack.append(iter(search_direct_deps(meet)))
return result
def copy_package(pkgname: str, dest: Path):
pkgfiles = pkg_files(pkgname)
if pkgfiles is None: return
dest_abs = dest.resolve()
for pkgfile in pkgfiles:
file_abs = (dest_abs / pkgfile).resolve()
if not file_abs.is_relative_to(dest_abs): continue
file_abs.parent.mkdir(parents=True, exist_ok=True)
file_abs.write_bytes(pkgfile.read_binary())
PROJ_DIST.mkdir(exist_ok=True)
config = read_yaml(HERE / f'{sys.argv[1]}.yaml')
main_name: str = config['main']
file_name = f'{main_name}_py{PYVER.major}{PYVER.minor}'
modules: list[str] = [main_name, *config['modules']]
print(f'递归查找包的依赖...')
packages = search_all_deps([Requirement(pkg.lower()) for pkg in config['site-packages']])
print(f'已找到依赖的包:{packages}')
module_paths = [get_module_path(name) for name in modules]
cache_path = PROJ_DIST / f'{file_name}.cache'
if cache_path.exists(): rmtree(cache_path)
cache_path.mkdir(exist_ok=True)
print('复制依赖的包到缓存目录...')
for pkg in packages: copy_package(pkg, cache_path)
ZipApp(
compressed=True,
cache_path=str(cache_path),
includes=','.join(str(p) for p in module_paths),
main=f'{main_name}.__main__',
output=str(PROJ_DIST / f'{file_name}.pyz'),
unzip='AUTO',
).build()
An example config file for this script:
main: mymod
modules:
- local_module
site-packages:
- ipython
- pyyaml
- pillow
- prompt-toolkit
- pycryptodome
- requests
- zipp
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.