Comments (14)
Is pants the only way to do this at the moment? I can't seem to find any documentation on it
from pex.
sadly yes but this should be a pretty straightforward PR
from pex.
If I understand it correctly, it would be a case of iterating through the combinations and creating resolvers for each, installing packages from those resolvers and then only after that, creating/freezing the pex file? I can't see where Pants is handling its multiplatform stuff
from pex.
@wickman is anybody working on this? I'm not very familiar with the pex codebase, but I would like to take a try at implementing this if nobody else is.
We're using pex at the Internet Archive to generate binaries for our ia
command-line tool (https://github.com/jjjake/internetarchive). I would like to add some dependencies that would require platform specific distributions, and it would be great if we could package everything into a single binary that could be used on multiple platforms.
Thanks for all the work on pex, it's been a big win for us at the Archive being able to easily package the ia
command-line tool as a single binary that works on many platforms. : )
from pex.
@jjjake I've not heard of anyone planning to tackle this in pex
anytime soon, so I'd say feel free to take a stab at it.
btw, per the earlier conversation on this thread here's the relevant code from pants
: https://github.com/pantsbuild/pants/blob/97c0bdba3eb39ee8dd4836a6547bed7e6cf937d6/src/python/pants/backend/python/python_chroot.py#L241
it'd be ideal if we could subsume this functionality directly into pex
at the library level (e.g. a pluralized platforms=
kwarg to pex.resolver.resolve
etc), such that pants
could utilize baked-in multi-platform support via the API and drop its own multi-platform logic.
from pex.
Awesome. Thank you for the details @kwlzn, those are helpful. I'll go ahead and take a stab at it.
from pex.
@kwlzn here's my first pass at making --platform/--interpreter multi-options: jjjake@d948fa9
Please let me know if I'm on the right track, and I'll work on cleaning things up, docstrings and tests for this change. Thanks!
from pex.
@jjjake assuming this is functional, the approach looks reasonable to me at a high level. I'll hold off on smaller/style comments until you have a pull request ready to go.
I'd recommend cutting a branch for your changes before you get much further which should lead to an easier initial PR - then once you have a PR up you can utilize our TravisCI integration for test results (and keep pushing updates from your branch to re-run CI as needed during iteration).
from pex.
Commit 572f7f1 including jjjake's is now passing travis tests (https://travis-ci.org/mzdanieltest/pex/builds/132036887), though I noticed cache_dir has a different meaning (it is a list instead of a string). In particular, my original pex call doesn't run:
pex -v --disable-cache --no-index -f dist -c sphinxserve -o pex/scripts/sphinxserve dist/*
and raises:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/bin/pex", line 11, in <module>
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pex/bin/pex.py", line 537, in main
options.cache_dir = make_relative_to_root(options.cache_dir)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pex/bin/pex.py", line 518, in make_relative_to_root
return os.path.normpath(path.format(pex_root=ENV.PEX_ROOT))
AttributeError: 'list' object has no attribute 'format'
What would be the correct call to create a multi interpreter and platform?
Do the the different binary wheels need to be in different directories?
from pex.
@mzdaniel this was actually a bug, fixed in ec63631 today. rebasing master should include this fix so you can continue testing.
btw, once you think your continuation of jjjake's change is ready to go, feel free to craft a new PR and tag that with the ready-for-review
label and I'll take a look.
from pex.
We have integrated pex with bazel for Heron compilation and we want to support multi-platform pex files for distribution. We are augmenting existing pex binary rules to take in platform option for generating these pex files. As I understand, the functionality of multi platform is with pants. Wondering if you could make this functionality available with pex and it will help us seamless integrate with bazel as well.
from pex.
@kramasamy I've landed support for this in master - should be able to cut a release tomorrow for you guys to begin using.
from pex.
@kramasamy release is out: https://pypi.python.org/pypi/pex/1.2.9
from pex.
from pex.
Related Issues (20)
- pex lock update: mechanism to remove projects from lockfile requirements HOT 7
- pex lock update: mechanism to change the version specifier if an existing requirement (or constraint?) HOT 5
- gmsh library not loaded in the packaged pex python binary HOT 21
- Documentation website is down/missing HOT 6
- lock update does not preserve "binary-ness" and errors out with artifact set changes HOT 6
- Support Pip's `--only-binary` / `--no-binary`. HOT 1
- Support syncing input requirements to a lock file more easily.
- Unify CLI options with Pip options when requirements files are parsed. HOT 2
- URL quoting regression when using `find-links` HOT 1
- Generating lockfiles fails with: unknown error (_ssl.c:3161) HOT 3
- Consider using uv as an optional alternate resolver. HOT 21
- Expose Pip's `--use-feature`. HOT 2
- How to build pex file with requirements.txt ? HOT 3
- Can we build the project without requirements.txt ? HOT 3
- lock create doesn't support tar.bz2 extensions HOT 6
- Implement more permissive `pex3 lock sync` locked resolve selection in multi-resolve locks.
- Trying to use pex to build a ML runtime. HOT 3
- unable to create pex with --no-build flag from lockfile HOT 2
- `--no-pre-install-wheels --requirements-pex .. --layout packed` gives `[Errno 21] Is a directory: '.../.deps/....whl'` errors HOT 3
- Incompatible Python version + `--complete-platforms ...` + `--lock ...` + empty cache gives seemingly-spurious "requires different Python" errors HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pex.