mamba-org / boa Goto Github PK

View Code? Open in Web Editor NEW

253.0 8.0 56.0 748 KB

The fast conda package builder, based on mamba

Home Page: https://boa-build.readthedocs.io/en/latest/

License: BSD 3-Clause "New" or "Revised" License

Python 99.80% Batchfile 0.07% Shell 0.13%

conda conda-packages mamba

boa's People

Contributors

Stargazers

Watchers

boa's Issues

Builds continue after solver failures

When the solver fails a message is printed like below however the build continues regardless:

MAMBA failed to solve specs

, _spcs,

for channels

%s

The reported errors are:

 ['local', 'conda-forge', 'defaults'] Encountered problems while solving.
Problem: package davix-0.6.7-hfd78c64_0 requires libxml2 >=2.9.8,<2.10.0a0, but none of the providers can be installed

check that run_constrained works fine

Cannot install due to missing README

The readme is used in the setup.py but doesn't exist here causing pip install git+https://github.com/TheSnakePit/boa.git to fail.

https://github.com/TheSnakePit/boa/blob/d9873c9e04eb3be2c4abf7a5ffa1dd2bfc5da228/setup.py#L20

Implement file list include / exclude & force_include sections

In conda-build, one can do split packages by specifying a "files" field.
In the files field, one can have a list of "globs" to include into the package.

However, there are some shortcomings: it's not easy to exclude files, and the file globs actually operate on the entire prefix and not just the file-diff.

I want to implement new sub-keys: include, exclude and force_include.

include operates on the newly added files and selects them by glob.
After that, files in exclude are subtracted from the included files.
with force_include, one can select files from the prefix that do not need to come from the newly added files (ie. the entire prefix). That should be avoided in nearly 100% of the time.

boa 0.3.2 fails to build and test noarch package

boa version 0.3.2 is failing to build & test a conda package when it's noarch: python.

Consider the following recipe for a simple dummy package which is noarch: python

package:
  name: foo
  version: "0.1.1"

build:
  noarch: python

requirements:
  build:
    - setuptools
  run:
    - python>=3.6,<3.7

test:     # we need whatever here just to force the creation of the test environment
  imports:
    - os
  requires:
    - pytest

Building the package above with:

mamba mambabuild .

gives the error message below (truncated):

...
INFO conda_build.utils:shutil_move_more_retrying(2091): shutil.move(work)=/Users/greis/conda/envs/boa032/conda-bld/foo_1610557324713/work, dest=/Users/greis/conda/envs/boa032/conda-bld/foo_1610557324713/work_moved_foo-0.1.1-py_0_noarch)
conda-forge/noarch       Using cache
Users/greis/conda/envs/b [====================] (00m:00s) Done
main/noarch              [====================] (00m:00s) No change
Users/greis/conda/envs/b [====================] (00m:00s) Done
Mamba failed to solve:
 - foo 0.1.1 py_0
 - python
 - pytest
 - python

with channels:
 - file:///Users/greis/conda/envs/boa032/conda-bld

The reported errors are:
⇟   Encountered problems while solving.
   Problem: nothing provides requested python
   Problem: nothing provides requested pytest
   Problem: nothing provides requested python
   Problem: nothing provides python >=3.6,<3.7 needed by foo-0.1.1-py_0

Expected behaviour

When I use boa=0.3.0 then it builds the noarch package without any problems. The boa=0.3.2 also builds non-noarch packages fine. @wolfv So I think it's a problem related to boa=0.3.2 and noarch: python packages.

environment

The relevant packages on my environment are:

boa                       0.3.2            py38h50d1736_1    conda-forge
conda                     4.9.2            py38hecd8cb5_0    main
conda-build               3.20.5                   py38_1    main
mamba                     0.7.6            py38he6f4b85_0    conda-forge
python                    3.8.5                h26836e1_1    main

Easy file-tests macros

Some ideas for easier writing tests. Currently people do stuff like

- test -f $PREFIX/include/mamba/test.hpp  # [unix]
- if not %LIBRARY_PREFIX%\include\mamba\test.hpp exit 1 # [win]

which is super error prone and hard to get right.

Ideas for "files" macros:

site-packages("mamba") -> checks that pythonX.X/site-packages/ exists and has contents (at least one init.py)
pymodule("mamba") -> checks that pythonX.X/site-packages/ exists and has contents (at least one init.py)
lib("libmamba", soversion="...") -> check that at least libmamba.so / libamamba.dylib is found in lib/, additionally check for so-versions that link to that, or for windows, check that bin/libmamba.dll and lib/libmamba.lib are there
include("mamba") -> check that include folder include/mamba/ exists
include("mamba/test.hpp") -> check that file include/mamba/test.hpp` exists in package
`bin("mamba")

I am unsure about the following ones:

cmake_find("mamba") -> make sure that find_package("mamba") would work
pkgconfig("mamba") -> make sure that pkgconfig mamba works

There are probably some good ideas to support other languages and their install prefixes. I don't know where R installs to?!

We can also use qualifiers to disallow any files outside those matched by the macros, except maybe files in /share or some other data-centric directory. That would give pretty strict control over the package content.

`conda mambabuild` fails with `--output-folder` flag

When I type:

conda mambabuild --output-folder output timm

I get the error:

Traceback (most recent call last):
  File "/home/jhoward/anaconda3/bin/conda-mambabuild", line 11, in <module>
    sys.exit(main())
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 91, in main
    variants=args["variants"],
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/conda_build/api.py", line 195, in build
    variants=variants
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/conda_build/build.py", line 3011, in build_tree
    test(pkg, config=metadata.config.copy(), stats=stats)
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/conda_build/build.py", line 2805, in test
    is_conda=metadata.name() == 'conda')
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/conda_build/environ.py", line 892, in create_env
    display_actions(actions, index)
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/conda/exports.py", line 236, in display_actions
    actions['LINK'] = [index[d] for d in actions['LINK']]
  File "/home/jhoward/anaconda3/lib/python3.7/site-packages/conda/exports.py", line 236, in <listcomp>
    actions['LINK'] = [index[d] for d in actions['LINK']]
KeyError: Dist(channel='local', dist_name='timm-0.3.2-py_0', name='timm', fmt='.tar.bz2', version='0.3.2', build_string='py_0', build_number=0, base_url=None, platform=None)

Using conda build instead works correctly.

mambabuild: support for anaconda tokens

Similar to mamba-org/mamba#370

Seems like a simple change to port, but I'm not sure about token leaks elsewhere in the code.

Index for all channels not refreshing on mambabuild

The index repodata cache is not being refreshed for all the conda channels because of the big ttl set on:

boa/boa/cli/mambabuild.py

Line 71 in fc9050d

context.local_repodata_ttl = 100000

I think this need to be configurable or, at least, much, much smaller.

@wolfv @marcelotrevisani - what do you think?

I searched at mamba's codebase and the ttl of the cache doesn't seem to be set to a high number there, so it's a problem only for mambabuild.

`conda mambabuild` does not respect passed channels

Greetings all,

I love how much fast mamba is over conda, but am having a little trouble getting it to respect conda overrides during the build process. conda mambabuild -c conda-forge <recipe> does not cause mamba to look at the conda-forge channel (it just continues to look at the default channels of the host environment). Is that expected?

Implement a REPL

When a recipe fails, it's a pretty difficult and annoying process to change the recipe, add patches, ...

We want to make this as smooth as possible. The goal is to implement a REPL when recipes fail, using prompt_toolkit:

open editor in workdir, modify the contents of workdir to interactively create patch files and add them to the recipe
change dependencies in the recipe.yaml file and the host / build environment are immediately updated & changed
change the build script in the recipe dir and re-run the recipe script
clean workdir / host easily to re-run recipe quickly
use ccache if possible?

missing repodata.json when using --output-folder on clean build directory

I found out that the error I have is not the same as #22 , it is triggered later on in the code execution.

Running conda mambabuild conda --output-folder build on a fresh boa installation or after conda build purge-all raises the following:

Traceback (most recent call last):
  File "/opt/conda/bin/conda-mambabuild", line 11, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 91, in main
    variants=args["variants"],
  File "/opt/conda/lib/python3.7/site-packages/conda_build/api.py", line 195, in build
    variants=variants
  File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 3013, in build_tree
    notest=notest,
  File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 2048, in build
    create_build_envs(top_level_pkg, notest)
  File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 1896, in create_build_envs
    channel_urls=tuple(m.config.channel_urls))
  File "/opt/conda/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 52, in mamba_get_install_actions
    solver.replace_channels()
  File "/opt/conda/lib/python3.7/site-packages/boa/core/solver.py", line 98, in replace_channels
    self.local_index = get_index(("local",), platform=self.platform, prepend=False)
  File "/opt/conda/lib/python3.7/site-packages/mamba/utils.py", line 74, in get_index
    is_downloaded = dlist.download(True)
RuntimeError: Download error (37) Couldn't read a file:// file [file:///opt/conda/conda-bld/noarch/repodata.json]

running without --output-folder works fine, and subsequent calls works also with it.

Is there a `conda debug` equivalent?

Greetings! I just discovered mamba/boa, and the performance improvements offered by mamba are AMAZING. Some of my recipes are not working out of the box with conda mambabuild, however; so is there a conda debug equivalent?

auto-discovery of recipes and dependencies

One can start conda-build in a folder, and it will try to find all recipes + outputs to build up a dependency tree.

We should have that in boa as well, especially for bootstrapping with boa-forge.

for static libraries, use a different libdir?

We could think about using a differnt libdir for static libraries. The benefit would be that pkgconfig files and cmake files could be re-generated and live in two places, once for the static library and once for the dynamic one.

E.g. CMAKE_INSTALL_LIBDIR=libstatic for cmake

Virtual packages not working in conda mambabuild

Getting the following error in trying to do a build with conda mambabuild:

The reported errors are:
   Problem: nothing provides __glibc >=2.17,<3.0.a0 needed by ucx-1.8.1+g6b29558-cuda11.0_0

I believe __glibc is a virtual package provided by conda.

If a minimal reproducer recipe would help just let me know and I'll throw something together

Conan tools look quite interesting

Might be worth to see if we can re-use some of this stuff.

https://docs.conan.io/en/latest/reference/tools.html

use correct channel priorities in solver

This should use the new API that we want in mamba: mamba-org/mamba#545

make sure tests are run

conda mambabuild supports only a single recipe

As the title says, e.g. --no-test is not forwarded to conda build, and thus tests are still executed although they shouldn't.

Implement activation scripts in intershell-language

Currently, activation scripts have to be written per-shell. This is quite fragile and difficult, as shells have some slightly different semantics, and one is seldomly an expert on many shells.
Also, support for e.g. powershell is quite weak (in conda-forge).

If we have a high-level language for shell operations we could make life easier for developers.

This coudl be a joint effort with the AShT project (http://github.com/regro/asht Abstract Shell Tree), and I started some experiments at https://github.com/wolfv/multisheller

use strict channel prio if configured

Implement parallel building

With the ROS rebuilds we could still save ~50% of the time if we would execute builds in parallel.

Lot's of time is taken by linking packages into an environment, and by running low-cpu stuff like cmake scripts.

We can create a graph of all recipes, and run all in parallel whenever possible.

quickly bail when patch doesn't apply

currently when a patch doesn't work, we continue.
The build fails at a later stage with 'no source found'

we should error earlier and deliver a concise error message.

Add handling of `pin_run_as_build`

This seems to be quite heavily used in conda-forge.

Cache not loaded

When trying to conda mambabuild . with vinca, I get the following error:

Traceback (most recent call last):
  File "/Users/tobias/opt/anaconda3/bin/conda-mambabuild", line 33, in <module>
    sys.exit(load_entry_point('boa', 'console_scripts', 'conda-mambabuild')())
  File "/Users/tobias/robot/boa/boa/cli/mambabuild.py", line 284, in main
    solver.replace_channels()
  File "/Users/tobias/robot/boa/boa/cli/mambabuild.py", line 151, in replace_channels
    cp = subdir.cache_path()
RuntimeError: Cache not loaded!

Any hints why that would be?

ModuleNotFoundError: No module named 'ruamel.yaml'

Experimenting with boa gives me this error (running from the project directory:

$ boa .
Traceback (most recent call last):
  File "/opt/conda/bin/boa", line 33, in <module>
    sys.exit(load_entry_point('boa', 'console_scripts', 'boa')())
  File "/opt/conda/bin/boa", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/conda/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/test/boa/boa/cli/render.py", line 2, in <module>
    from ruamel.yaml import YAML
ModuleNotFoundError: No module named 'ruamel.yaml'

Installing ruamel and ruamel_yaml does not help (adapted from the readme):

mamba install conda-build colorama pip ruamel ruamel_yaml -c conda-forge

additional test macro: check file, check directory

It would be good to have additional functionality in boa / conda-build to easily check for the existence of a file / directory in the output.

Currently this is pretty cumbersome with test -f ... etc.

config: translate `boost_cpp` -> `boost-cpp`

We have to replace underscores with dashes so that names match :(

Handle strong run_exports in recipe

Currently it's not correctly handled which leads to issues downstream.

Properly close files in mamba / boa

After building many recipes in one boa session, lsof shows many open file handles for the same json and solv cache files. We might have to properly close those in replace_channels.

Colored output and filter host prefix on stdout

conda-build uses a super long host placeholder prefix but it messes up the build logs.

Could be interesting to filter it on the stdout stream and replace with or something like that.

Metadata needs get_used_vars?

Used when printing skip reason.

+    def get_used_vars(self):
+        # TODO this is used when skipping (variants!)
+        print("get_used_vars not implemented, can't print variants")
+        return []

Properly remove files on Win

https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexa?redirectedfrom=MSDN

Add ccache

We could try to (somehow) support ccache to speed up builds.
This is especially interesting when developing recipes, as well as when building outputs with many overlapping compiled files (e.g. static & dynamic builds of libraries).

Support variant files in mambabuild

One of the useful features of mamba is that its resolver errors are much more readable. In conda-forge PRs, we often run into problems where a resolution due to the constraints given by the global pinnings file cannot be resolved. Here it would be useful to take conda mambabuild to get a different error using conda mambabuild -m .ci_support/osx_r_base3.6target_platformosx-64.yaml recipe.

recipes as tests, test-mode

It would be great if recipes could also act as first-class "test" scripts that people use in CI to test their packages. That would allow users to only use one source of truth for testing & releasing.

To make this happen, I propose boa test

with boa test, boa would skip all logic wrt to relocation and packaging, and just build the recipe, then install the test requirements and run the things from the test section.

Hashing differences

There are slight differences in the contents in get_hash_dependencies to compute the build string:

conda includes CONDA_BUILD_SYSROOT
conda includes channel_targets
conda seems to not include build deps except compilers?
if only target_platform in the variant, conda disregards that

conda mambabuild fails with "KeyError: Dist"

Using conda mambabuild isn't working for me anymore since at least today as I get the same error for all (Python-based) recipes:

Script started on Wed Nov 11 12:20:48 2020
�[1m�[7m%�[27m�[1m�[0m                                                                                                                                                                                                           
 
�]0;uwe@Uwes-QuantCo-MacBook-Pro: ~/Development/dagster-feedstock�
�[0m�[27m�[24m�[J�[1m�[31m�[39m�[0m�[1m�[34muwe�[39m�[0m@Uwes-QuantCo-MacBook-Pro �[1m~/Development/dagster-feedstock �[0m�[35m(�[39mgit�[35m)�[33m-�[35m[�[32mrebuild-pypy-0-1_h998514�[35m]�[39m % �[K�[?1h�=�[?2004h�[7mconda mambabuild -m .ci_support/osx_64_python3.6.____73_pypy.yaml recipe�[27m�[72D�[27mc�[27mo�[27mn�[27md�[27ma�[27m �[27mm�[27ma�[27mm�[27mb�[27ma�[27mb�[27mu�[27mi�[27ml�[27md�[27m �[27m-�[27mm�[27m �[27m.�[27mc�[27mi�[27m_�[27ms�[27mu�[27mp�[27mp�[27mo�[27mr�[27mt�[27m/�[27mo�[27ms�[27mx�[27m_�[27m6�[27m4�[27m_�[27mp�[27my�[27mt�[27mh�[27mo�[27mn�[27m3�[27m.�[27m6�[27m.�[27m_�[27m_�[27m_�[27m_�[27m7�[27m3�[27m_�[27mp�[27my�[27mp�[27my�[27m.�[27my�[27ma�[27mm�[27ml�[27m �[27mr�[27me�[27mc�[27mi�[27mp�[27me�[?1l�>�[?2004l
�]0;uwe@Uwes-QuantCo-MacBook-Pro: conda mambabuild -m .ci_support/osx_64_python3.6.____73_pypy.yaml recipe�Updating build index: /Users/uwe/miniconda3/conda-bld

conda-forge/osx-64       Using cache
conda-forge/noarch       Using cache
�[2K
XXX/osx-64      [>                   ] (--:--) No change
�[1A�[2K
XXX/osx-64      [====================] (00m:00s) No change
�[1A�[0K�[2K
XXX/osx-64      [====================] (00m:00s) No change
�[2K
XXX/noarch      [>                   ] (--:--) No change
�[1A�[2K
XXX/noarch      [====================] (00m:00s) No change
�[1A�[0K�[2K
XXX/noarch      [====================] (00m:00s) No change
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /Users/uwe/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /Users/uwe/conda_build_config.yaml
Adding in variants from /Users/uwe/Development/dagster-feedstock/.ci_support/osx_64_python3.6.____73_pypy.yaml
INFO:conda_build.variants:Adding in variants from /Users/uwe/Development/dagster-feedstock/.ci_support/osx_64_python3.6.____73_pypy.yaml
Attempting to finalize metadata for dagster
INFO:conda_build.metadata:Attempting to finalize metadata for dagster
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
Transaction

  Prefix: /Users/uwe/miniconda3

  Nothing to remove

�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
Transaction

  Prefix: /Users/uwe/miniconda3

  Updating specs:

   - pip
   - python=3.6[build=*_73_pypy]


  Package            Version  Build           Channel                  Size
─────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────

�[32m  bzip2          �[00m      1.0.8  haf1e3a3_3      conda-forge/osx-64�[32m     Cached�[00m
�[32m  ca-certificates�[00m  2020.11.8  h033912b_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  certifi        �[00m  2020.6.20  py36he57f5f5_2  conda-forge/osx-64�[32m     Cached�[00m
�[32m  expat          �[00m      2.2.9  hb1e8313_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  gdbm           �[00m       1.18  h8a0c380_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  libcxx         �[00m     11.0.0  h439d374_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  libffi         �[00m      3.2.1  hb1e8313_1007   conda-forge/osx-64�[32m     Cached�[00m
�[32m  ncurses        �[00m        6.2  h2e338ed_3      conda-forge/osx-64�[32m     Cached�[00m
�[32m  openssl        �[00m     1.1.1h  haf1e3a3_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  pip            �[00m     20.2.4  py_0            conda-forge/noarch�[32m     Cached�[00m
�[32m  pypy3.6        �[00m      7.3.2  h9f31681_2      conda-forge/osx-64      26 MB
�[32m  python         �[00m      3.6.9  3_73_pypy       conda-forge/osx-64�[32m     Cached�[00m
�[32m  python_abi     �[00m        3.6  1_pypy36_pp73   conda-forge/osx-64�[32m     Cached�[00m
�[32m  readline       �[00m        8.0  h0678c8f_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  setuptools     �[00m     49.6.0  py36he57f5f5_2  conda-forge/osx-64�[32m     Cached�[00m
�[32m  sqlite         �[00m     3.33.0  h960bd1c_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  tk             �[00m     8.6.10  hb0a8c7a_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  wheel          �[00m     0.35.1  pyh9f0ad1d_0    conda-forge/noarch�[32m     Cached�[00m
�[32m  xz             �[00m      5.2.5  haf1e3a3_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  zlib           �[00m     1.2.11  h7795811_1010   conda-forge/osx-64�[32m     Cached�[00m

  Summary:

  Install: 20 packages

  Total download: 26 MB

─────────────────────────────────────────────────────────────────────────────

�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
Transaction

  Prefix: /Users/uwe/miniconda3

  Updating specs:

   - certifi==2020.6.20=py36he57f5f5_2
   - ncurses=6.2[build=h2e338ed_3]
   - gdbm=1.18[build=h8a0c380_2]
   - expat==2.2.9=hb1e8313_2
   - openssl==1.1.1h=haf1e3a3_0
   - setuptools==49.6.0=py36he57f5f5_2
   - ca-certificates==2020.11.8=h033912b_0
   - wheel==0.35.1=pyh9f0ad1d_0
   - readline=8.0[build=h0678c8f_2]
   - python_abi=3.6[build=1_pypy36_pp73]
   - pypy3.6==7.3.2=h9f31681_2
   - tk==8.6.10=hb0a8c7a_1
   - libcxx==11.0.0=h439d374_0
   - xz==5.2.5=haf1e3a3_1
   - libffi==3.2.1=hb1e8313_1007
   - sqlite==3.33.0=h960bd1c_1
   - bzip2==1.0.8=haf1e3a3_3
   - python==3.6.9=3_73_pypy
   - pip==20.2.4=py_0
   - zlib==1.2.11=h7795811_1010


  Package            Version  Build           Channel                  Size
─────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────

�[32m  bzip2          �[00m      1.0.8  haf1e3a3_3      conda-forge/osx-64�[32m     Cached�[00m
�[32m  ca-certificates�[00m  2020.11.8  h033912b_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  certifi        �[00m  2020.6.20  py36he57f5f5_2  conda-forge/osx-64�[32m     Cached�[00m
�[32m  expat          �[00m      2.2.9  hb1e8313_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  gdbm           �[00m       1.18  h8a0c380_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  libcxx         �[00m     11.0.0  h439d374_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  libffi         �[00m      3.2.1  hb1e8313_1007   conda-forge/osx-64�[32m     Cached�[00m
�[32m  ncurses        �[00m        6.2  h2e338ed_3      conda-forge/osx-64�[32m     Cached�[00m
�[32m  openssl        �[00m     1.1.1h  haf1e3a3_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  pip            �[00m     20.2.4  py_0            conda-forge/noarch�[32m     Cached�[00m
�[32m  pypy3.6        �[00m      7.3.2  h9f31681_2      conda-forge/osx-64      26 MB
�[32m  python         �[00m      3.6.9  3_73_pypy       conda-forge/osx-64�[32m     Cached�[00m
�[32m  python_abi     �[00m        3.6  1_pypy36_pp73   conda-forge/osx-64�[32m     Cached�[00m
�[32m  readline       �[00m        8.0  h0678c8f_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  setuptools     �[00m     49.6.0  py36he57f5f5_2  conda-forge/osx-64�[32m     Cached�[00m
�[32m  sqlite         �[00m     3.33.0  h960bd1c_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  tk             �[00m     8.6.10  hb0a8c7a_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  wheel          �[00m     0.35.1  pyh9f0ad1d_0    conda-forge/noarch�[32m     Cached�[00m
�[32m  xz             �[00m      5.2.5  haf1e3a3_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  zlib           �[00m     1.2.11  h7795811_1010   conda-forge/osx-64�[32m     Cached�[00m

  Summary:

  Install: 20 packages

  Total download: 26 MB

─────────────────────────────────────────────────────────────────────────────

Adding .* to spec 'docstring_parser 0.7.1' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING:conda_build.utils:Adding .* to spec 'docstring_parser 0.7.1' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING conda_build.utils:ensure_valid_spec(1862): Adding .* to spec 'docstring_parser 0.7.1' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
Adding .* to spec 'pendulum 1.4.4' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING:conda_build.utils:Adding .* to spec 'pendulum 1.4.4' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING conda_build.utils:ensure_valid_spec(1862): Adding .* to spec 'pendulum 1.4.4' to ensure satisfiability.  Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead.  See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
BUILD START: ['dagster-0.9.18-py36h5dafb3c_2.tar.bz2']
�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [>                   ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[2K
Users/uwe/miniconda3/con [=>                  ] (--:--) Finalizing...
�[1A�[2K
Users/uwe/miniconda3/con [=>                  ] (--:--) Done
�[1A�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
�[1A�[0K�[2K
Users/uwe/miniconda3/con [====================] (00m:00s) Done
Transaction

  Prefix: /Users/uwe/miniconda3

  Updating specs:

   - bzip2==1.0.8=haf1e3a3_3
   - ca-certificates==2020.11.8=h033912b_0
   - libcxx==11.0.0=h439d374_0
   - ncurses=6.2[build=h2e338ed_3]
   - xz==5.2.5=haf1e3a3_1
   - zlib==1.2.11=h7795811_1010
   - expat==2.2.9=hb1e8313_2
   - libffi==3.2.1=hb1e8313_1007
   - openssl==1.1.1h=haf1e3a3_0
   - readline=8.0[build=h0678c8f_2]
   - tk==8.6.10=hb0a8c7a_1
   - gdbm=1.18[build=h8a0c380_2]
   - sqlite==3.33.0=h960bd1c_1
   - pypy3.6==7.3.2=h9f31681_2
   - python_abi=3.6[build=1_pypy36_pp73]
   - python==3.6.9=3_73_pypy
   - certifi==2020.6.20=py36he57f5f5_2
   - wheel==0.35.1=pyh9f0ad1d_0
   - setuptools==49.6.0=py36he57f5f5_2
   - pip==20.2.4=py_0


  Package            Version  Build           Channel                  Size
─────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────

�[32m  bzip2          �[00m      1.0.8  haf1e3a3_3      conda-forge/osx-64�[32m     Cached�[00m
�[32m  ca-certificates�[00m  2020.11.8  h033912b_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  certifi        �[00m  2020.6.20  py36he57f5f5_2  conda-forge/osx-64�[32m     Cached�[00m
�[32m  expat          �[00m      2.2.9  hb1e8313_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  gdbm           �[00m       1.18  h8a0c380_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  libcxx         �[00m     11.0.0  h439d374_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  libffi         �[00m      3.2.1  hb1e8313_1007   conda-forge/osx-64�[32m     Cached�[00m
�[32m  ncurses        �[00m        6.2  h2e338ed_3      conda-forge/osx-64�[32m     Cached�[00m
�[32m  openssl        �[00m     1.1.1h  haf1e3a3_0      conda-forge/osx-64�[32m     Cached�[00m
�[32m  pip            �[00m     20.2.4  py_0            conda-forge/noarch�[32m     Cached�[00m
�[32m  pypy3.6        �[00m      7.3.2  h9f31681_2      conda-forge/osx-64      26 MB
�[32m  python         �[00m      3.6.9  3_73_pypy       conda-forge/osx-64�[32m     Cached�[00m
�[32m  python_abi     �[00m        3.6  1_pypy36_pp73   conda-forge/osx-64�[32m     Cached�[00m
�[32m  readline       �[00m        8.0  h0678c8f_2      conda-forge/osx-64�[32m     Cached�[00m
�[32m  setuptools     �[00m     49.6.0  py36he57f5f5_2  conda-forge/osx-64�[32m     Cached�[00m
�[32m  sqlite         �[00m     3.33.0  h960bd1c_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  tk             �[00m     8.6.10  hb0a8c7a_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  wheel          �[00m     0.35.1  pyh9f0ad1d_0    conda-forge/noarch�[32m     Cached�[00m
�[32m  xz             �[00m      5.2.5  haf1e3a3_1      conda-forge/osx-64�[32m     Cached�[00m
�[32m  zlib           �[00m     1.2.11  h7795811_1010   conda-forge/osx-64�[32m     Cached�[00m

  Summary:

  Install: 20 packages

  Total download: 26 MB

─────────────────────────────────────────────────────────────────────────────

Traceback (most recent call last):
  File "/Users/uwe/miniconda3/bin/conda-mambabuild", line 11, in <module>
    sys.exit(main())
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 89, in main
    variants=args["variants"],
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/api.py", line 195, in build
    variants=variants
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/build.py", line 3013, in build_tree
    notest=notest,
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/build.py", line 2048, in build
    create_build_envs(top_level_pkg, notest)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/build.py", line 1882, in create_build_envs
    is_conda=m.name() == 'conda')
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/environ.py", line 907, in create_env
    display_actions(actions, index)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda/exports.py", line 236, in display_actions
    actions['LINK'] = [index[d] for d in actions['LINK']]
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda/exports.py", line 236, in <listcomp>
    actions['LINK'] = [index[d] for d in actions['LINK']]
KeyError: Dist(channel='conda-forge', dist_name='ca-certificates-2020.11.8-h033912b_0', name='ca-certificates', fmt='.tar.bz2', version='2020.11.8', build_string='h033912b_0', build_number=0, base_url=None, platform=None)
  17,17s user 1,25s system 90% cpu 20,454 total
�[1m�[7m%�[27m�[1m�[0m                                                                                                                                                                                                           
 
�]0;uwe@Uwes-QuantCo-MacBook-Pro: ~/Development/dagster-feedstock�
�[0m�[27m�[24m�[J�[1m�[31m1 �[39m�[0m�[1m�[34muwe�[39m�[0m@Uwes-QuantCo-MacBook-Pro �[1m~/Development/dagster-feedstock �[0m�[35m(�[39mgit�[35m)�[33m-�[35m[�[32mrebuild-pypy-0-1_h998514�[35m]�[39m % �[K�[103C:(�[105D�[?1h�=�[?2004h�[?2004l

Script done on Wed Nov 11 12:21:19 2020

test activation & deactivation scripts

A test could be to run the activation script, print some (expected) env vars, then run deactivation and make sure that the environment is mostly unchanged.

export env variables from variants

I think we're missing $target_platform and other variant-exported env vars.

After failed build, print $PREFIX and other helpful commands

Would be cool to print a table like:

$PREFIX = /home/user/miniconda/conda-bld/...

Try building again with /home/user/miniconda/conda-bld/sompkg_1234/conda_build.sh

translator from new-style to old-style recipe

this would be useful for backward compatibility with conda-build

some work being done on an universal recipe reader is here: https://github.com/marcelotrevisani/souschef

Multiple test sections in a single output?

One of the issues with having a single tests sections in a single output is that adding test-only dependencies can result in broken packages sneaking out. For example:

test:
    requires:
      - pip
      - pytest
    imports:
      - example
    commands:
      - pip check
      - pytest -vvv

Possible failures are:

If example depends on pkg_resources the pip requirement will provide setuptools at test time but not runtime.
If example depends on pluggy the pytest requirement will satisfy the requirement at test time but not runtime.

I've seen this quite a few times both in Python packages and more complicated things. A possible solution would be to have three separate test sections and environments like:

tests:
    # Basic import tests
    - imports:
        - example
    # Check dependencies
    - requires:
        - pip
      commands:
        - pip check
    # Unit tests
    - requires:
        - pytest
      commands:
        - pytest -vvv

Better yet, some of them could be marked as optional/slow to allow them to be skipped unless --with-optional-tests is passed to conda build/boa.

Handle missing repodata.json

conda build purge-all removes various folders, including conda-bld/noarch which leads to this error in conda mambabuild:

Traceback (most recent call last):
  File "/home/peter/miniconda3/bin/conda-mambabuild", line 8, in <module>
    sys.exit(main())
  File "/home/peter/miniconda3/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 278, in main
    solver.replace_channels()
  File "/home/peter/miniconda3/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 149, in replace_channels
    self.local_index = get_index(("local",), platform=self.platform, prepend=False)
  File "/home/peter/miniconda3/lib/python3.7/site-packages/boa/cli/mambabuild.py", line 111, in get_index
    is_downloaded = dlist.download(True)
RuntimeError: Download error (37) Couldn't read a file:// file [file:///home/peter/miniconda3/conda-bld/noarch/repodata.json]

Specification X.Y is interpreted as X.Y.0

This might be another issue you know about but I'll dump it here never the less :)

Version requirements like X.Y are interpreted as X.Y.0 instead of X.Y.*. I've seen this with python 3.7 using 3.7.0 and numpy 1.14 using 1.14.0. These versions avoid the issue in #2 but result in linker errors due to the pinnings not being quite right.

boa test sometimes has too strict specs

when testing, boa get's specs from conda and they are

python 3.7

which is bad because that is python 3.7.0 implicitly with mamba. So we need to fix that somehow.

mambabuild crashes when dependencies exists in local ~/conda-bld

Consider the following minimal recipe:

package:
  name: foo
  version: 1.0.0

requirements:
  run:
    - zlib >=1.2.11,<1.3.0a0

test:
  commands:
    - ls    # anything here just to force creation of the test env

I'm using zlib as a run dependency there just because it's a dependency that doesn't bring anything else. But just imagine that this is a dependency that is also built by your team/organization and the packages will exist in the local conda-bld channel, in the CI build agents.

Create a conda-bld directory in your HOME, to serve as local channel, and copy the zlib package from your package cache to the subdir of your platform. For example, executing the following commands:

mkdir $HOME/conda-bld
mkdir $HOME/conda-bld/osx-64
mkdir $HOME/conda-bld/noarch
cp $HOME/miniconda3/pkgs/zlib-1.2.11-h1de35cc_3.tar.bz2 $HOME/conda-bld/osx-64
conda index $HOME/conda-bld

Now let's try to build the recipe for foo as specified above with conda-build and boa.

When running conda build . the build succeeds. You can observe that

output using `conda build`

The following NEW packages will be INSTALLED:

    foo:  1.0.0-0           local
    zlib: 1.2.11-h1de35cc_3 main

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

Note that the zlib package was fetched from main (not from local). So far, so good, the build works.

However if you build the foo package with boa by invoking mamba mambabuild ., you get a failure:

output using `mamba mambabuild`

  File "/Users/greis/miniconda3/envs/mamba/lib/python3.8/site-packages/conda/exports.py", line 236, in <listcomp>
    actions['LINK'] = [index[d] for d in actions['LINK']]
KeyError: Dist(channel='local', dist_name='zlib-1.2.11-h1de35cc_3', name='zlib', fmt='.tar.bz2', version='1.2.11', build_string='h1de35cc_3', build_number=3, base_url=None, platform=None)

Note below that the zlib was found in the local conda-bld channel created above, but failed to fetch it later.

I've done some investigation and the problem is that the action plan found the zlib to exist in the local channel, but when crossing that with the index the same Dist object is not there. It seems that this is related to what boa is doing when monkey patching the function:

conda_build.environ.get_install_actions = mamba_get_install_actions

And then inside the function mamba_get_install_actions() it calls solver.replace_channels().

We can either fix this by making the behaviour to be like the one in conda (which leads to the action to pick up the dependency from the external or non-local channels) or we can make it the index to also consider the local channel.

@wolfv can you please share your thoughts?

We'd like to use mamba to build our projects, we're past the trial phase. But now when I moved to adopt in the CI, the simple existence of the conda-bld folder in the home directory of the user account that runs the builds causes mambabuild to crash. We can't remove that conda-bld folder because other projects won't start using mamba just yet and will still point their configurations there.

environment

The most relevant packages on my environment are (shortlist):

boa                       0.3.0            py38h50d1736_1    conda-forge
conda                     4.9.2            py38hecd8cb5_0    main
conda-build               3.20.5                   py38_1    main
conda-package-handling    1.7.2            py38h22f3db7_0    main
mamba                     0.7.6            py38he6f4b85_0    conda-forge
python                    3.8.5                h26836e1_1    main

Support cross-compilation "natively"

Cross-compilation is already possible with the toolchains in conda-forge, but I don't know exactly how to do it (i.e. the "user interface" is not very obvious).
I think BinaryBuilder.jl does a really good job there... It would be great to have similar native & automatic cross-compilation features in boa.

Working on different envs

When working in a different env than base, boa doesn't find mamba

mamba-org / boa Goto Github PK

boa's People

Contributors

Stargazers

Watchers

Forkers

boa's Issues

Expected behaviour

environment

Ideas for "files" macros:

I am unsure about the following ones:

output using conda build

output using mamba mambabuild

environment

Recommend Projects

Recommend Topics

Recommend Org

output using `conda build`

output using `mamba mambabuild`