ofek / pyapp Goto Github PK
View Code? Open in Web Editor NEWRuntime installer for Python applications
Home Page: https://ofek.dev/pyapp/
Runtime installer for Python applications
Home Page: https://ofek.dev/pyapp/
When embedding a local tar of a python distribution, I get the following error
--- stderr
thread 'main' panicked at build.rs:565:9:
Unable to determine format for distribution source:
(yes, distribution_source
is an empty string, I checked).
This occurs when providing PYAPP_PROJECT_PATH
(embedding a wheel), PYAPP_DISTRIBUTION_PATH
, and PYAPP_EXEC_SPEC
. Removing PYAPP_DISTRIBUTION_PATH
, works but python is not bundled along with the binary.
I have an example here.
full output of make build-bugged
$ make build-bugged
poetry build -f wheel
Building pyapp-bug (0.1.0)
- Building wheel
- Built pyapp_bug-0.1.0-py3-none-any.whl
cd pyapp-latest && \
PYAPP_PROJECT_PATH=../dist/pyapp_bug-0.1.0-py3-none-any.whl \
PYAPP_DISTRIBUTION_PATH=./python-3.10.13.tar.gz \
PYAPP_EXEC_SPEC=pyapp_bug.entry:main \
cargo build --release
Compiling pyapp v0.13.0 (/home/nmay/cur/pyapp_bug/pyapp-latest)
error: failed to run custom build command for `pyapp v0.13.0 (/home/nmay/cur/pyapp_bug/pyapp-latest)`
Caused by:
process didn't exit successfully: `/home/nmay/cur/pyapp_bug/pyapp-latest/target/release/build/pyapp-fa2006654aeb24f9/build-script-build` (exit status: 101)
--- stdout
cargo:rustc-env=PYAPP_PROJECT_DEPENDENCY_FILE=
cargo:rustc-env=PYAPP__PROJECT_DEPENDENCY_FILE_NAME=
cargo:rustc-env=PYAPP_PROJECT_NAME=pyapp-bug
cargo:rustc-env=PYAPP_PROJECT_VERSION=0.1.0
cargo:rustc-env=PYAPP__PROJECT_EMBED_FILE_NAME=pyapp_bug-0.1.0-py3-none-any.whl
cargo:rustc-env=PYAPP_DISTRIBUTION_SOURCE=
cargo:rustc-env=PYAPP__DISTRIBUTION_ID=12080488350626410958
--- stderr
thread 'main' panicked at build.rs:565:9:
Unable to determine format for distribution source:
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
make: *** [Makefile:13: build-bugged] Error 101
I built the binary from outside container then copied to container by using a a simple Dockerfile.
# syntax=docker/dockerfile:1
FROM debian:12-slim
WORKDIR /app
COPY ./dist/app/my-app-bin ./
RUN /app/my-app-bin self restore
ENTRYPOINT ["/app/my-app-bin"]
But docker build failed with:
STEP 5/6: RUN /app/my-app-bin self restore
/bin/sh: 1: /app/my-app-bin: not found
Error: building at STEP "RUN /app/my-app-bin self restore": while running runtime: exit status 127
Exec to container to call bin directly:
root@9c0da96a3e1d:/app# /app/my-app-bin self restore
bash: /app/my-app-bin: cannot execute: required file not found
root@9c0da96a3e1d:/app# ls
my-app-bin
Is it the correct way to build docker container with pyapp? How can I make it work? Thanks.
Hi,
big fans of pyapp
here. In its current form it lends itself very well for this use case: distribute lightweight, self-bootstrapping python environments based on lockfiles ๐ช
Icing on the cake would be an additional management command equivalent to the hatch env shell
command: <pyapp-binary> self shell
. Unfortunately no Rust skills beyond compiling existing projects to offer.
> cargo install pyapp --quiet --root out
> mv out/bin/pyapp cowsay && chmod +x cowsay
> cowsay -t hi
App: cowsay
Version: LATEST
Python: CPython 11 (x86_64 for Windows)
The app you are trying to use is not installed.
Do you want [I]nstall it with above config, [M]odify setup config or [A]bort installation?
Currently i can only use app.exe self restore
to re-create the distribution, which itself is also removing the old one.
Can we get a management command like app.exe self remove
to only delete the installation files?
I'm not familiar with the Rust toolchain and have a Python Poetry-based project. The requirements are captured in a poetry.lock
file, although poetry can export requirements.txt
if required. How would I go about using this really interesting project to package my app for Windows desktop deployment?
After my last issue I tried to prepackage everything into a custom python distribution but I cannot seem to get it to work. As a test I want my binary to run python -m http.server
:
curl -LO https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.11.3%2B20230507-x86_64_v2-unknown-linux-gnu-install_only.tar.gz
PYAPP_SKIP_INSTALL=1 PYAPP_FULL_ISOLATION=1 PYAPP_DISTRIBUTION_FORMAT="tar|gzip" PYAPP_DISTRIBUTION_PATH=`pwd`/cpython-3.11.3%2B20230507-x86_64_v2-unknown-linux-gnu-install_only.tar.gz PYAPP_EXEC_MODULE=http.server PYAPP_PROJECT_NAME=test PYAPP_PROJECT_VERSION=1 cargo install pyapp --force --root /tmp/
This build fine and utilizes PYAPP_DISTRIBUTION_PATH
to reuse the already downloaded distribution. Running the binary results in:
/tmp/bin/pyapp
Error: project execution failed, consider restoring from scratch
Caused by:
No such file or directory (os error 2)
any hints?
๐
As discussed in DM, it'd be great if I built a PyApp and tried running it in parallel if the bootstrapping didn't race.
Great project, thanks for the wonderful work! I tried creating a PyApp
package for a project that requires additional dependencies. With pip
, the installation would simply be:
pip install my_package[extra]
However, I don't seem to be able to figure out how I can pass the [extra]
to PyApp
directly. Is it necessary in this case to create a requirements.txt
file and pass that via PYAPP_PROJECT_DEPENDENCY_FILE
or can the extra deps be passed directly?
Thanks!
Compiling pyapp v0.13.0
error: failed to run custom build command for `pyapp v0.13.0`
Caused by:
process didn't exit successfully: `C:\WINDOWS\TEMP\cargo-installsnqYmf\release\build\pyapp-73fe803f2bd031c2\build-script-build` (exit code: 101)
--- stdout
cargo:rustc-env=PYAPP_PROJECT_NAME=cowsay
cargo:rustc-env=PYAPP_PROJECT_VERSION=6.0
cargo:rustc-env=PYAPP_PROJECT_DEPENDENCY_FILE=
cargo:rustc-env=PYAPP__PROJECT_DEPENDENCY_FILE_NAME=
cargo:rustc-env=PYAPP__PROJECT_EMBED_FILE_NAME=
--- stderr
thread 'main' panicked at C:\Users\User\.cargo\registry\src\index.crates.io-6f17d22bba15001f\pyapp-0.13.0\build.rs:353:5:
No default distribution source found
Python version: 3.11
Platform: windows
Architecture: x86_64
ABI: gnu
Variant: shared
I think wrong ABI selected.
one less env set
When running PyApp packaged GUI on Windows, a console window is generally open which is not really the wanted behavior. This is not the case on Linux, don't know right how it is on macos...
In order run a GUI without a console windows on Windows, two things would be required as far as I can see it:
#![windows_subsystem = "windows"]
to main.rs
, see RFC 1665pythonw.exe
instead of python.exe
when running.For 1 I currently don't see a good way of using an environmental variable. One possible solution would be to add
[features]
gui = []
to Cargo.toml
. This way, the windows_subsystem = "windows"
top-level crate attribute could simply be activated during compilation with cargo build --features gui
.
I'll prepare a draft PR that implements this, plus the pythonw.exe
usage for running the program. Let me know what you think, this is just an idea at this point... the additions in the PR should make it easier to see hopefully.
The documentation states that python 3.7 through 3.11 are supported and that it defaults to the latest stable minor version of CPython if PYAPP_PYTHON_VERSION
is not set.
However, the docs seem to have diverged from the code in the following sense:
v0.12.0
, pyapp
supports python 3.12PYAPP_PYTHON_VERSION
is unset, pyapp
defaults to 3.11
I propose to update the docs and set the default python to 3.12. Along with those changes, the python versions could also be upgraded to 20240224.
Let me know what you think, I'd be happy to implement this.
The available python versions provide 3.7
without a PYAPP_DISTRIBUTION_VARIANT
. However, if the OS is linux
and arch is x86_64
, this variant cannot be accessed at all. In this case, if the PYAPP_DISTRIBUTION_VARIABLE
is empty, it defaults to v3
-> which is not available. However, there seems to be no setting that the variable can be set to in order to access the variant ""
that is the list.
This seems to be only an issue for 3.7
.
Some possible fixes:
3.7
supportPYAPP_DISTRIBUTION_VARIANT
variable when 3.7
is selected, e.g., by changing the else if in the if statement in line 311 in build.rs
to:
if variant.is_empty() {
if selected_platform == "windows" {
variant = "shared".to_string();
} else if selected_platform == "linux" && selected_arch == "x86_64" && selected_python_version != "3.7" {
variant = "v3".to_string();
}
};
In addition, I think an overview page in the docs on what pythons are available in what configuration would be helpful. This could be auto-generated from the update_distributions.py
script for example.
Again, happy to contribute these changes :)
Hi there,
pyapp
looks really great by I somewhat fail to get it to run. Is there any way to debug the build process? I am invoking pyapp
like this:
PYAPP_PROJECT_VERSION=1.0 PYAPP_PROJECT_NAME=csi_plugin_nfs PYAPP_DISTRIBUTION_EMBED=1 PYAPP_PROJECT_DEPENDENCY_FILE=/home/florian/sources/csi-plugin-nfs/requirements.txt cargo install pyapp --force --root /tmp/test
with a requirements file (see below) to be able to control what ends up in the binary. Yet when I try to run it I get:
/home/florian/.local/share/pyapp/csi-plugin-nfs/3476349063995752034/1.0/bin/python3: No module named csi_plugin_nfs
Also, is there any way to allow pip to only install wheels and no sdists?
The requirements file:
# This file is @generated by PDM.
# Please do not edit it manually.
grpc-interceptor==0.15.1 \
--hash=sha256:1cc52c34b0d7ff34512fb7780742ecda37bf3caa18ecc5f33f09b4f74e96b276 \
--hash=sha256:3efadbc9aead272ac7a360c75c4bd96233094c9a5192dbb51c6156246bd64ba0
grpcio==1.54.0 \
--hash=sha256:02000b005bc8b72ff50c477b6431e8886b29961159e8b8d03c00b3dd9139baed \
--hash=sha256:031bbd26656e0739e4b2c81c172155fb26e274b8d0312d67aefc730bcba915b6 \
--hash=sha256:1209d6b002b26e939e4c8ea37a3d5b4028eb9555394ea69fb1adbd4b61a10bb8 \
--hash=sha256:125ed35aa3868efa82eabffece6264bf638cfdc9f0cd58ddb17936684aafd0f8 \
--hash=sha256:1382bc499af92901c2240c4d540c74eae8a671e4fe9839bfeefdfcc3a106b5e2 \
--hash=sha256:16bca8092dd994f2864fdab278ae052fad4913f36f35238b2dd11af2d55a87db \
--hash=sha256:1c59d899ee7160638613a452f9a4931de22623e7ba17897d8e3e348c2e9d8d0b \
--hash=sha256:1d109df30641d050e009105f9c9ca5a35d01e34d2ee2a4e9c0984d392fd6d704 \
--hash=sha256:1fa7d6ddd33abbd3c8b3d7d07c56c40ea3d1891ce3cd2aa9fa73105ed5331866 \
--hash=sha256:21c4a1aae861748d6393a3ff7867473996c139a77f90326d9f4104bebb22d8b8 \
--hash=sha256:224166f06ccdaf884bf35690bf4272997c1405de3035d61384ccb5b25a4c1ca8 \
--hash=sha256:2262bd3512ba9e9f0e91d287393df6f33c18999317de45629b7bd46c40f16ba9 \
--hash=sha256:2585b3c294631a39b33f9f967a59b0fad23b1a71a212eba6bc1e3ca6e6eec9ee \
--hash=sha256:27fb030a4589d2536daec5ff5ba2a128f4f155149efab578fe2de2cb21596d3d \
--hash=sha256:30fbbce11ffeb4f9f91c13fe04899aaf3e9a81708bedf267bf447596b95df26b \
--hash=sha256:3930669c9e6f08a2eed824738c3d5699d11cd47a0ecc13b68ed11595710b1133 \
--hash=sha256:3b170e441e91e4f321e46d3cc95a01cb307a4596da54aca59eb78ab0fc03754d \
--hash=sha256:3db71c6f1ab688d8dfc102271cedc9828beac335a3a4372ec54b8bf11b43fd29 \
--hash=sha256:48cb7af77238ba16c77879009003f6b22c23425e5ee59cb2c4c103ec040638a5 \
--hash=sha256:49eace8ea55fbc42c733defbda1e4feb6d3844ecd875b01bb8b923709e0f5ec8 \
--hash=sha256:533eaf5b2a79a3c6f35cbd6a095ae99cac7f4f9c0e08bdcf86c130efd3c32adf \
--hash=sha256:5942a3e05630e1ef5b7b5752e5da6582460a2e4431dae603de89fc45f9ec5aa9 \
--hash=sha256:62117486460c83acd3b5d85c12edd5fe20a374630475388cfc89829831d3eb79 \
--hash=sha256:650f5f2c9ab1275b4006707411bb6d6bc927886874a287661c3c6f332d4c068b \
--hash=sha256:6dc1e2c9ac292c9a484ef900c568ccb2d6b4dfe26dfa0163d5bc815bb836c78d \
--hash=sha256:73c238ef6e4b64272df7eec976bb016c73d3ab5a6c7e9cd906ab700523d312f3 \
--hash=sha256:775a2f70501370e5ba54e1ee3464413bff9bd85bd9a0b25c989698c44a6fb52f \
--hash=sha256:860fcd6db7dce80d0a673a1cc898ce6bc3d4783d195bbe0e911bf8a62c93ff3f \
--hash=sha256:87f47bf9520bba4083d65ab911f8f4c0ac3efa8241993edd74c8dd08ae87552f \
--hash=sha256:960b176e0bb2b4afeaa1cd2002db1e82ae54c9b6e27ea93570a42316524e77cf \
--hash=sha256:a7caf553ccaf715ec05b28c9b2ab2ee3fdb4036626d779aa09cf7cbf54b71445 \
--hash=sha256:a947d5298a0bbdd4d15671024bf33e2b7da79a70de600ed29ba7e0fef0539ebb \
--hash=sha256:a97b0d01ae595c997c1d9d8249e2d2da829c2d8a4bdc29bb8f76c11a94915c9a \
--hash=sha256:b7655f809e3420f80ce3bf89737169a9dce73238af594049754a1128132c0da4 \
--hash=sha256:c33744d0d1a7322da445c0fe726ea6d4e3ef2dfb0539eadf23dce366f52f546c \
--hash=sha256:c55a9cf5cba80fb88c850915c865b8ed78d5e46e1f2ec1b27692f3eaaf0dca7e \
--hash=sha256:d2f62fb1c914a038921677cfa536d645cb80e3dd07dc4859a3c92d75407b90a5 \
--hash=sha256:d8ae6e0df3a608e99ee1acafaafd7db0830106394d54571c1ece57f650124ce9 \
--hash=sha256:e355ee9da9c1c03f174efea59292b17a95e0b7b4d7d2a389265f731a9887d5a9 \
--hash=sha256:e3e526062c690517b42bba66ffe38aaf8bc99a180a78212e7b22baa86902f690 \
--hash=sha256:eb0807323572642ab73fd86fe53d88d843ce617dd1ddf430351ad0759809a0ae \
--hash=sha256:ebff0738be0499d7db74d20dca9f22a7b27deae31e1bf92ea44924fd69eb6251 \
--hash=sha256:ed36e854449ff6c2f8ee145f94851fe171298e1e793f44d4f672c4a0d78064e7 \
--hash=sha256:ed3d458ded32ff3a58f157b60cc140c88f7ac8c506a1c567b2a9ee8a2fd2ce54 \
--hash=sha256:f4a7dca8ccd8023d916b900aa3c626f1bd181bd5b70159479b142f957ff420e4
grpcio-reflection==1.54.0 \
--hash=sha256:804326e1add80050cab248107d28f226be58ec49d5a2d08f14a150d8a2621678 \
--hash=sha256:9de46150bba3c039035c5f573a9348f1c747d3149b1fa946b351da7c0e0733d8
protobuf==4.22.3 \
--hash=sha256:13233ee2b9d3bd9a5f216c1fa2c321cd564b93d8f2e4f521a85b585447747997 \
--hash=sha256:23452f2fdea754a8251d0fc88c0317735ae47217e0d27bf330a30eec2848811a \
--hash=sha256:52f0a78141078077cfe15fe333ac3e3a077420b9a3f5d1bf9b5fe9d286b4d881 \
--hash=sha256:70659847ee57a5262a65954538088a1d72dfc3e9882695cab9f0c54ffe71663b \
--hash=sha256:7760730063329d42a9d4c4573b804289b738d4931e363ffbe684716b796bde51 \
--hash=sha256:7cf56e31907c532e460bb62010a513408e6cdf5b03fb2611e4b67ed398ad046d \
--hash=sha256:8b54f56d13ae4a3ec140076c9d937221f887c8f64954673d46f63751209e839a \
--hash=sha256:d14fc1a41d1a1909998e8aff7e80d2a7ae14772c4a70e4bf7db8a36690b54425 \
--hash=sha256:d4b66266965598ff4c291416be429cef7989d8fae88b55b62095a2331511b3fa \
--hash=sha256:e0e630d8e6a79f48c557cd1835865b593d0547dce221c66ed1b827de59c66c97 \
--hash=sha256:ecae944c6c2ce50dda6bf76ef5496196aeb1b85acb95df5843cd812615ec4b61 \
--hash=sha256:f08aa300b67f1c012100d8eb62d47129e53d1150f4469fd78a29fa3cb68c66f2 \
--hash=sha256:f2f4710543abec186aee332d6852ef5ae7ce2e9e807a3da570f36de5a732d88e
csi_plugin_nfs-0.6.1-py3-none-any.whl --hash=sha256:f420920f27201389b7da35a604c4a44080e21a9cd5ab424f213187bcd6134ce2
Hi, thanks for your work with PyApp, it seems like a great way to distribute an App without requiring the user to deal with Python & Pip Dependencies!
When I tried using it unfortunately it fails after downloading the python distribution with the following message:
Error: unable to move /tmp/.tmpm0yLxY/394365705109672478 to /home/user/.cache/pyapp/distributions/394365705109672478
Caused by:
Invalid cross-device link (os error 18)
I suspect that PyApp downloads the distribution to /tmp... Folder and tries to move it into the /home user folder. This however fails, since I have my /home directory encrypted, thus /tmp and /home are on different mountpoints.
I have created a PR that fixes this problem: #36
After built with pyapp, calling pyapp self restore will fail because it could not install a dependency: hdbscan
gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64-v3 -fPIC -fPIC -I/root/.local/share/pyapp/myapp/2930268585031877112/2.0.5/python/include/python3.11 -I/tmp/pip-build-env-8_vyixci/overlay/lib/python3.11/site-packages/numpy/core/include -c hdbscan/_hdbscan_tree.c -o build/temp.linux-x86_64-cpython-311/hdbscan/_hdbscan_tree.o
Please note that the -march=x86-64-v3
, which is not correct, should be x86-64
. Without packaging with pyapp, I can install with normal pip without issue. Another issue is it does not detect gcc, but will require clang
FROM rust:1-debian
ENV PYAPP... # similar to cowsay, expose pip
WORKDIR /app
RUN cargo install pyapp --root /app
RUN pyapp pip install hdbscan # <-- fail
Workaround
FROM rust:1-debian
ENV PYAPP... # similar to cowsay, expose pip
WORKDIR /app
RUN cargo install pyapp --root /app
RUN CC=gcc CFLAGS="-march $(uname -m | tr '_' '-')" pyapp pip install hdbscan # <-- ok
Hello,
im on Ubuntu 22.04 and just created with the tutorial a binary executable for cowsay.
If i try to run that binary in a centos:7 docker container it crashes and shows
./cowsay: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.29' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./cowsay)
i assume that the glibc version on centos is too low for the container it says
glibc-common-2.17-317.el7.x86_64
glibc-2.17-317.el7.x86_64
and ldd cowsay
gives
./cowsay: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.29' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./cowsay)
./cowsay: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./cowsay)
linux-vdso.so.1 => (0x00007fffaa1dd000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007c29302b9000)
libm.so.6 => /lib64/libm.so.6 (0x00007c292ffb7000)
libc.so.6 => /lib64/libc.so.6 (0x00007c292fbe9000)
/lib64/ld-linux-x86-64.so.2 (0x00007c2930894000)
Can we target different libc versions with pyapp?
I sady have not enough knowledge of rust
can this lib work with local runnig streamlit with access to USB devices and outer .ini files?
https://streamlit.io/
I encountered a build failure on Windows that only happens with PYAPP_PROJECT_PATH.
I wanted to check out pyapp to see if it's usable in my day job, so I tried it on one of my hobby projects.
What works:
Setting up the build via PYAPP_PROJECT_NAME=autodigipick
and PYAPP_PROJECT_VERSION=1.1.1
It grabs the files from PyPI and executes without problem.
What doesn't work:
Downloading the whl directly from PyPI into the pyapp project root to see how it handles proprietary projects that aren't on any index.
Setting PYAPP_PROJECT_PATH=autodigipick-1.1.1-py3-none-any.whl
and running cargo build --release
leads to the build failure below [1].
The panic message looks suspicious, almost as if the last character in name
given to normalize_project_name()
contains a carriage return character.
From what I can gather from the stacktrace, it looks like pyapp is trying read the name from the METADATA file contained in the whl. Since autodigipick was built on Windows, the METADATA file created during the build process has CRLF line endings.
The Regex in set_project_from_metadata()
includes the \r in the capture and causes the Regex in normalize_project_name()
to not match and subsequently panic.
If my deduction is correct, this is less a build failure on Windows, but a build failure with whl files built on Windows. But I'll keep the issue title vague for now since I'm not deep enough in this project to accurately deduct the root cause.
[1] Console output:
D:\git\pyapp>cargo build --release
Compiling pyapp v0.15.0 (D:\git\pyapp)
error: failed to run custom build command for `pyapp v0.15.0 (D:\git\pyapp)`
Caused by:
process didn't exit successfully: `D:\git\pyapp\target\release\build\pyapp-9c9a7185155e1e53\build-script-build` (exit code: 101)
--- stdout
cargo:rustc-env=PYAPP_PROJECT_DEPENDENCY_FILE=
cargo:rustc-env=PYAPP__PROJECT_DEPENDENCY_FILE_NAME=
--- stderr
thread 'main' panicked at build.rs:259:9:
`; must only contain ASCII letters/digits, underscores, hyphens, and periods, and must begin and end with ASCII letters/digits.
stack backtrace:
0: 0x7ff65ae66e32 - std::sys_common::backtrace::_print::impl$0::fmt
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\sys_common\backtrace.rs:44
1: 0x7ff65ae8118d - core::fmt::rt::Argument::fmt
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\core\src\fmt\rt.rs:142
2: 0x7ff65ae8118d - core::fmt::write
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\core\src\fmt\mod.rs:1120
3: 0x7ff65ae63301 - std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\io\mod.rs:1810
4: 0x7ff65ae66c5a - std::sys_common::backtrace::_print
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\sys_common\backtrace.rs:47
5: 0x7ff65ae66c5a - std::sys_common::backtrace::print
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\sys_common\backtrace.rs:34
6: 0x7ff65ae68ea9 - std::panicking::default_hook::closure$1
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:272
7: 0x7ff65ae68b65 - std::panicking::default_hook
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:292
8: 0x7ff65ae693d4 - std::panicking::rust_panic_with_hook
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:779
9: 0x7ff65ae692a9 - std::panicking::begin_panic_handler::closure$0
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:657
10: 0x7ff65ae674d9 - std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::closure_env$0,never$>
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\sys_common\backtrace.rs:171
11: 0x7ff65ae68f72 - std::panicking::begin_panic_handler
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:645
12: 0x7ff65af77b27 - core::panicking::panic_fmt
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\core\src\panicking.rs:72
13: 0x7ff65a66e42a - build_script_build::normalize_project_name
at D:\git\pyapp\build.rs:259
14: 0x7ff65a66fac2 - build_script_build::set_project_from_metadata
at D:\git\pyapp\build.rs:378
15: 0x7ff65a670c8b - build_script_build::set_project
at D:\git\pyapp\build.rs:443
16: 0x7ff65a675d99 - build_script_build::main
at D:\git\pyapp\build.rs:931
17: 0x7ff65a67710b - core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce\library\core\src\ops\function.rs:250
18: 0x7ff65a67b02e - core::hint::black_box
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce\library\core\src\hint.rs:286
19: 0x7ff65a67b02e - std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> >
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce\library\std\src\sys_common\backtrace.rs:155
20: 0x7ff65a67b1f1 - std::rt::lang_start::closure$0<tuple$<> >
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce\library\std\src\rt.rs:166
21: 0x7ff65ae5ec72 - std::rt::lang_start_internal::closure$2
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\rt.rs:148
22: 0x7ff65ae5ec72 - std::panicking::try::do_call
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:552
23: 0x7ff65ae5ec72 - std::panicking::try
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panicking.rs:516
24: 0x7ff65ae5ec72 - std::panic::catch_unwind
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\panic.rs:142
25: 0x7ff65ae5ec72 - std::rt::lang_start_internal
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library\std\src\rt.rs:148
26: 0x7ff65a67b1ca - std::rt::lang_start<tuple$<> >
at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce\library\std\src\rt.rs:165
27: 0x7ff65a675e09 - main
28: 0x7ff65af5b650 - invoke_main
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
29: 0x7ff65af5b650 - __scrt_common_main_seh
at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
30: 0x7ffabe437614 - BaseThreadInitThunk
31: 0x7ffabf0626a1 - RtlUserThreadStart
Hi,
Would it be possible to add an option to name rge executable based on the project name rather than pyapp.exe?
Congratulations by the way, really useful took.
Hey Ofek,
really great job you're doing here!
I'd really like to get PyApp running and got first promising results already.
I'm trying to bundle an app that both has scripts
and gui-scripts
defined.
Unfortunately PyApp does not support selecting the GUI entrypoint in the binary target option scripts
.
I can somehow workaround it and define my GUI entrypoints under [project.scripts]
, since luckily even like this no console window is shown, but that's no how it's supposed to be.
I tried setting the script option to an empty list and define the execution mode by setting one of PYAPP_EXEC_*
, but this is not working, since "empty list" is considered as "default to all entries of [project.scripts]
". So afaik there is currently no chance of defining a custom execution mode, or is it??
I just noticed it does not matter if i set PYAPP_IS_GUI
to 0
or 1
, the executable is always starting a pythonw.exe
, contrary to your documentation here.
Just as a reference, this is part of my config:
[project.scripts]
myapp_cli = "myapp.cli:main"
myapp-cli = "myapp.cli:main"
[project.gui-scripts]
myapp-gui = "myapp.cli:main"
[tool.hatch.envs.default.env-vars]
PIP_INDEX_URL = "https://..."
# PYAPP
PYAPP_PIP_EXTRA_ARGS = "--index-url https://..."
PYAPP_IS_GUI = "0"
PYAPP_DISTRIBUTION_EMBED = "false"
PYAPP_FULL_ISOLATION = "true"
[tool.hatch.build.targets.binary]
scripts = ["myapp-cli"]
Then i'm building with hatch build -t binary
.
While using the standalone Hatch binaries, I noticed that in some circumstances the exit code is incorrect. For more information, please check following issue: juftin/hatch-pip-compile#66
I'm testing pyapp
and I'm liking it a lot. But in my testings it's always needed to have network to run the application for the first time. I tested with PYAPP_FULL_ISOLATION=1
and PYAPP_DISTRIBUTION_EMBED=1
this way:
PYAPP_DISTRIBUTION_FORMAT="tar|zstd" PYAPP_PROJECT_PATH="$PWD/dist/bioscaffolds-0.1.0.post213.dev0+8e76bb1-py3-none-any.whl" PYAPP_EXEC_MODULE="BioScaffolds" PYAPP_PROJECT_NAME="BioScaffolds" PYAPP_PROJECT_VERSION="0.1.0.post216.dev0+3d0ad9c" PYAPP_DISTRIBUTION_EMBED=1 PYAPP_PROJECT_DEPENDENCY_FILE="$PWD/requirements.txt" PYAPP_FULL_ISOLATION=1 PYAPP_IS_GUI=1 cargo install pyapp --force --root dist
And when I try to run without network:
dist/bin/pyapp
Error: unable to unpack to /var/home/thiago/.local/share/pyapp/bioscaffolds/15033190860216094598/0.1.0.post213.dev0+8e76bb1
failed to iterate over archive
I tested with PYAPP_SKIP_INSTALL=1
and I had the same problem.
Is it possible to make the executable no need network on runtime?
I've created a fork that adds a PYAPP_EXEC_NOTEBOOK execution option. It works in the same way as PYAPP_EXEC_SCRIPT but a .ipynb file needs to be specified instead of a .py file. The -m notebook [file.ipynb]
command is called at runtime.
Happy to submit a pull request if it's a feature you want to add.
Fork: https://github.com/quinnhornblow/pyapp
Example repo: https://github.com/quinnhornblow/pyapp_notebook
I did some testing of the new PYAPP_EXEC_SCRIPT option(with PYAPP_EXEC_SCRIPT=~/bin/PWXtract/bin/main.py --force --root ~/bin/PWXtract) . It works really well except for one thing.
When running a script like this the path is changed and in my test case gives /home/pwb/.cache/pyapp/scripts/11235863200305506371/main.py rather than /home/pwb/bin/PWXtract/bin/main.py
Is there any way of disabling the caching (or otherwise) so that the original path is retained? Which would also work after moving the full directory of code files and pyapp binary to a new path?
Exporting the path of the pyapp binary to an environmental path on execution would also work as a workaround.
Thanks a lot for making this by the way :)
This looks potentially very useful for a project I'm working on where I essentially need to have my code as a totally portable directory which includes a python runtime in addition to my python code and lots of database drivers etc that can run in an offline environment.
I basically do an install on my online computer and then move the resulting directory to potentially offline machines for the actual running of the code.
I did a bit of testing but couldn't quite get it to work for me. I did as follows
export PYAPP_PROJECT_NAME=pwetl
export PYAPP_PROJECT_VERSION=0.1
export PYAPP_PYTHON_VERSION=3.10
export PYAPP_FULL_ISOLATION=1
export PYAPP_DISTRIBUTION_EMBED=1
export PYAPP_SKIP_INSTALL=1
cargo build --release
cargo install pyapp --force --root ~/bin/PWETL
Everyting seems to work with a pyapp executable created in ~/bin/PWETL/bin/ but I cant seem to figure out how to run a local python file with a relative path (needed since the directory will need to be moved around).
Is this possible, and which PYAPP_EXEC_MODULE / PYAPP_EXEC_SPEC / PYAPP_EXEC_CODE setting should I use for running ~/bin/PWETL/bin/main.py when the pyapp executable is in the same directory?
I tried I few ways but couldn't find one that worked...
To avoid having to fire up an HTTP server for the build when combining source
with skip_install
and embed
(i.e. a custom dist in which the project has been pre-installed). Essentially, this would allow pyapp
to work the same way as pyinstaller single-file bundles, but with the distribution being persisted on disk.
Hi, I think the pyapp compiled everytime CI run even no code change and I have enabled cache-to, cache-from in docker/build-push-action
. Is there a way for us to cache the pyapp build?
Hi @ofek, thanks for the great tool!
I've come up with a solution proposal to a known problem I'm facing, and I would like to know your thoughts on it and if it's possible to implement it on Pyapp
I've tried to be as detailed as possible, and I'm open to any questions or suggestions, thanks!
I've hidden the R&D Process on this comment to only what matters below:
I have a pretty standard Monorepo structure that provides a main package for all the other Projects but don't know them
A project refers to the monorepo using path dependencies, and they even might refer to other projects as well
Note: I'm doing this to separate dependencies e.g. not all projects need pytorch
It all works nicely under development mode.. until I want to build a release of any project
In the past, I have implemented convoluted solutions using Pyinstaller or Nuitka which ended up working to a certain extent but wasn't ideal (long story), so I decided to give Pyapp a try
As I saw somewhere, Python wheels aren't standardized for path dependenciesyet?, so whenever building a pyproject.toml
, the wheel won't be installable on other machines as the builder's local path is hardcoded
I don't really want to upload the code to PyPI as it is very specific to my use case, much like other monorepo opinions; even if that was the use case, Poetry can't have a versioned dependency on the main section and a path dependency on the dev section simultaneously of the same package
Ultimately, this yields either spaghetti solutions or the lack of it
I've spent two days of intensive digging through the documentation and issues everywhere, trying many build backends such as Poetry, Hatch, PDM and proposed Poetry plugins or solutions, but ultimately I couldn't get it to work. Raw Pyapp was the closest I gotI know it's used in Hatch!
I honestly don't remember much of what I tried yesterday, but I can say this wasn't ideal as including the packages as sdist isn't "safe" and annoying to define the glob imports, also the monorepo package isn't on the subpath of the projects, and Poetry fails
Long story short, I zipped the Poetry's Virtual Environment and set the proper relative paths on Pyapp variables for the executables, and used the full isolated mode, skip install.
It fails as the Python included there is a symlink to system Python. Setting poetry.virtualenvs.options.always-copy
to true
didn't do it as well(Consider this as a bug report? To embed some proper Python distribution on top of a local one?)
I'm not a fan of this solution as yours fetching and installation of the Python distribution feels more reliable and arguably universal
I ported the pyprojects.toml
to Hatch syntax and force-included the main package Broken
under ../../Broken
to the wheel. The embedding I'm using is PYAPP_PROJECT_PATH
as the built wheel
This failed as I didn't "inherit" the dependencies of the main package, the Virtual Environment contained properly ShaderFlow
and Broken
package, but not the dependencies of Broken
(the monorepo root's package)
This solution feels non ideal as I had to unset safety flags on Hatch, like the allow direct references and, well, including some other package on the wheel
After all the digging, I think this could be solved by the following:
dev-dependencies
on the pyproject.toml
of the project:[tool.poetry.dependencies]
python = ">=3.10,<3.13"
moderngl = "^5.8.2"
# ...
[tool.poetry.dev-dependencies]
broken = {path="../../", develop=true}
Building a wheel for this project won't include the broken
package, but it's ok
This isn't something you can implement on Pyapp, but a process users would need to define on their own
A pseudo code / implementation would be something like this (I didn't run nor test the logic):
from pathlib import Path
from dotmap import DotMap
import toml
def build_projects(path: Path, found: Set[Path]=None):
path = Path(path).resolve()
# Initialize empty set
found = found or set()
# Skip if already found
if path in found:
return
# Skip if no pyproject.toml exists
if not (path/"pyproject.toml").exists():
return
# Load pyproject.toml dictionary
pyproject = DotMap(toml.reads((path/"pyproject.toml").read_text()))
# Iterate and find all path dependencies
for name, dependency in pyproject.tool.poetry["dev-dependencies"].items():
# Find only path= dictionaries
if isinstance(data, str):
continue
if not dependency.path:
continue
# Dependency is a path
dependency = Path(data.path).resolve()
found.add(dependency)
# Build the wheel
with pushd(dependency):
subprocess.run(["poetry", "build", "--format", "wheel"])
# Recursively find wheels
wheels(dependency, found)
return found
# Build all wheels
projects = build_projects(Path.cwd())
# We can now get the wheels from the projects
wheels = [next(project.glob("dist/*.whl")) for project in projects]
Why we need all of this? For the next step and the proposed solution
We still use the main project's wheel on PYAPP_PROJECT_PATH
settings when building, and we would include the other wheels on a new PYAPP_LOCAL_DEPENDENCIES
setting or other name in your preference
# Include other local dependencies on the building step
os.environ["PYAPP_LOCAL_DEPENDENCIES"] = ":".join(wheels)
# Compile the project
subprocess.run(["pyapp", ...])
When Pyapp is installing the Virtual Environment, it would install the main project's wheel and the other local wheels as well
By using the Path Dependencies projects as development dependencies, we:
By installing the wheel of the main project and all other Path Dependencies wheels, we:
importlib
My intuition says that this would work very great !
HI,
I'm using MacOS and I created a custom project and installed it and run it fine at first. I then went to uninstall it by deleting $HOME/Library/Caches/pyapp and I tried to re-install it again but it complains:
Error: project execution failed, consider restoring from scratch
Caused by:
No such file or directory (os error 2)
How do I restore from scratch?
Thanks
Can it be used to encrypt python projects?
I think this would be a great idea, but I don't know how to implement it.
Hi there,
I came across in this project in the search of a way to distribute Python report builder scripts to my non-programmer colleagues.
So far I've been using pyinstaller to build one-file executables that are placed on a (slow, non-local) network share. Adding a few "heavier" dependencies (like pandas, etc.) causes the file size to balloon in size and slow down startup times to a crawl. Other solutions like having users install the executable locally cause issues by users staying on outdated versions. I want there to be one source of truth and only one way to start it.
I really like the approach of one executable installing and "caching" itself locally on the user's machine and then just delegate the call to the cache for faster startup times after the first run.
However, I have a hit a few road blocks while testing pyapp for this purpose:
So I would love to have a way to embed all the data, but without actually embedding it. Instead, the executable itself should be as tiny as possible to speed up startup times and delegate the call to the local installation. Only if the local installation is not available (or the management interface is called), it should read and map a blob file placed nearby that contains the all the code and data required for the full operation of pyapp.
It would look something like this:
///network/share/directory/
- create_XYZ_report.exe <-- tiny executable with the bare essentials to locate the local cache
- create_XYZ_report.blob <-- all the code and data required to deploy for the first run
Thanks a lot for developing this project! I am very excited to try it.
I wanted to use it with a simple example project using poetry. I am trying to do it on Windows and have the following powershell file (build_with_pyapp.ps1).
# Export poetry requirements to requirements.txt
poetry export -f requirements.txt --output requirements.txt
# Now set environment variables properly in powershell
$env:PYAPP_PROJECT_VERSION="0.1.0"
$env:PYAPP_PROJECT_NAME="pyappexample"
$env:PYAPP_PROJECT_DEPENDENCY_FILE="./requirements.txt"
$env:CARGO_TARGET_DIR="./target" # For speeding up future builds
$env:PYAPP_EXEC_SCRIPT="./pyappexample/circle.py"
cargo install pyapp
Unfortunately I am getting the following error:
[<compiling stuff...>]
error: failed to run custom build command for `pyapp v0.13.0`
Caused by:
process didn't exit successfully: `C:\Users\hanne\Documents\Programme\pyappexample\./target\release\build\pyapp-7c9fbca43121c7ae\build-script-build` (exit code: 101)
--- stdout
cargo:rustc-env=PYAPP_PROJECT_NAME=pyappexample
cargo:rustc-env=PYAPP_PROJECT_VERSION=0.1.0
--- stderr
thread 'main' panicked at C:\Users\hanne\.cargo\registry\src\index.crates.io-6f17d22bba15001f\pyapp-0.13.0\build.rs:389:9:
Dependency file is not a file: ./requirements.txt
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile `pyapp v0.13.0`, intermediate artifacts can be found at `C:\Users\hanne\Documents\Programme\pyappexample\./target`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
(pyappexample-py3.10) PS C:\Users\hanne\Documents\Programme\pyappexample>
I am executing the powershell script from the root directory, where my requirements.txt file is located.
Have a great day!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.