conda-forge / staged-recipes Goto Github PK

This has happened a few times lately. We really need to straighten this out so we get proper failures pre-merge not failed deployments post-merge. Even if we look carefully, we may miss something the linter doesn't and we don't want to be doing a post-merge scramble to fix everything.

I have a recipe that I am looking to submit that has compiled extensions which links to blas and lapack. Does anyone have experience building package which depend on these or have any suggestions on how get these to build in a portable manner? I'm only looking to support Linux and OS X with the package which hopefully makes the process a bit easier.

@jakirkham @ocefpaf
Feodor Fitsner (founder of AppVeyor) reached me, and after a short conversation, he suggested to bump CPU speed for Conda-Forge projects. He asks for an account which is used for Conda-Forge, so he can apply the changes. Is it conda-forge? (I will forward the information via e-mail)

We all love conda and there are many communities that build awesome packages that are easy to use. I would like to see more exchange between these communities to finally share more build-scripts, to develop one best-practice guide and finally to have channels that can be used together without breaking recipes - a list of trusted channels with similar guidelines.

For example the bioconda community - specialised on bioinformatic software. They have some very nice guides how to develop packages, they are reviewing and bulk-patches recipes if there are new features in conda to make the overall experience even better.
ping @johanneskoester, @daler and @chapmanb from BioConda fame

Omnia has a lot of cheminformatic software and a nice build-box based on phusion/holy-build-box-64 + CUDA and AMD APP SDK.
ping @kyleabeauchamp, @jchodera

With conda-forge there is now a new one and it would be great to get all interested people together to join forces here and don't replicate our recipes or copy them from one channel to the other just to make them compatible.

Another point is that we probably want to move recipes to default at some point and deliver our work back to Continuum - so that we can benefit from each other.

I can imagine that we all form a group of trusted communities and channels and activate them by default in our unified build-box - or we have one giant community channel. All this I would like to discuss with everyone that is interested and come up with a plan how to make this happen :)

What do you all think about this?
As a next step I would like to create a doodle to find a meeting data where at least one representative from all communities can participate.

Many thanks to Continuum Analytics for there continues support and the awesome development behind scientific python and this package manager.
ping @jakirkham @msarahan

Appears AppVeyor has stopped working on staged-recipes. Am seeing error messages like this when trying to restart anything.

Cannot read last commit information from GitHub repository

See the following related issues.

• http://help.appveyor.com/discussions/problems/197-cannot-read-last-commit-information-from-github-repository
• http://help.appveyor.com/discussions/problems/1369-cannot-read-last-commit-information-from-github-repository-nikolayitopenjudgesystem-unauthorized

@pelson is this related to the latest conda-smithy?

See https://travis-ci.org/conda-forge/staged-recipes/jobs/115055485

Repository created, please edit conda-forge.yml to configure the upload channels
and afterwards call 'conda smithy register-github'
usage: a tool to help create, administer and manage feedstocks.
       [-h]
       {init,register-github,register-ci,regenerate,recipe-lint,rerender} ...
a tool to help create, administer and manage feedstocks.: error: invalid choice: 'github-create' (choose from 'init', 'register-github', 'register-ci', 'regenerate', 'recipe-lint', 'rerender')
Traceback (most recent call last):
  File ".CI/create_feedstocks.py", line 118, in <module>
    subprocess.check_call(['conda', 'smithy', 'github-create', feedstock_dir] + owner_info)
  File "/Users/travis/miniconda/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda', 'smithy', 'github-create', '/var/folders/gw/_2jq29095y7b__wtby9dg_5h0000gn/T/tmp_pPzpc__feedstocks/autoconf-feedstock', '--organization', 'conda-forge']' returned non-zero exit status 2

There is quite a bit of boiler plate that goes into recipes. It is a pain making sure it is right for bother reviewers and reviewees (?). We should move towards automating this by using all means at our disposal (e.g. jinja, scripts, metapackages, etc.). The hope is that most of this (if not all) of this could be removed. What I would like us to do here is figure out what and how. One step in this direction would be taking a hard look at this script by @183amir, which converts PyPI packages into conformant recipes.

cc @msarahan @ocefpaf @pelson

Currently we are rate limited here. So, wait a bit before merging anything. Let's try restarting in ~20min of this issue being opened.

haversine 0.4.5
Calculate the distance between 2 points on Earth.

Calculate the distance (in km or in miles) between two points on Earth, located by their latitude and longitude.

https://github.com/mapado/haversine
https://pypi.python.org/pypi/haversine

Fits nicely with basemap and cartopy :-)

For an example of this problem, see PR and this build.

Travis CI doesn't like it if something doesn't produce output for 10 mins. While this can be the sign of a hang, there are perfectly reasonable scenarios where it isn't a hang, but a silent long running command. Unfortunately, there is no simple way to change the time limit (unless this has changed recently 🙏).

One way to workaround this is to use travis_wait, which is essentially a hack that writes to disk periodically. We could try to run conda build through this, but IIRC it gobbles up stdout, which is really unhelpful not too mention this would need to go in the CI and thus affect all recipes in the staging area, which I really don't like. If we have to add it to feedstocks that need it as a workaround, that could be ok until we come up with something better. Just as long as we verify that this let's the build complete correctly and is not merely hiding some more serious problem. In other words, have submitters try and then remove before merging.

Alternatively, we could come up with some background process that does a similar thing to travis_wait.

Any other thoughts on this problem? Other potential solutions?

Currently we are rate limited here. So, wait a bit before merging anything. Let's try restarting in ~30min-1hr of this issue being opened.

Is there a way to set up a feedstock to upload to a custom channel in addition to uploading to the conda-forge channel?

@patricksnape Would you be kind and add a vlfeat recipe? you can also add me as a maintainer.

As reported at https://github.com/pelson/Obvious-CI/issues/58, conda has some open issues regarding non-ASCII text on Python 2 which don't appear anymore on Python 3:

https://github.com/conda/conda/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+unicode+OR+ascii

Specifically, conda/conda#1486 appeared to bite us after submitting #214.

Would it be possible to use Miniconda 3 for staged-recipes?

Hi everyone,

Are we going to have opencv3 in conda-forge? If yes, could someone who has experience please do it?
Maybe @jakirkham you can do it?

Could we save some CI time by using nomkl? What would be the consequences?

There are a couple of unrelated issues that have come up with non-nonsensical failures on the titular platform when no other platforms fail. In particular, these are #395 and conda-forge/xonsh-feedstock#7.

@pelson @melund @ocefpaf @jakirkham @msarahan @patricksnape

This is happening with bob.extension. Looks like it is an issue with AppVeyor (again). See the log for more details.

IRIS is a reasoning tool (http://iris-reasoner.org/demo)
I can run a simple demo, for example:
q('a').
p(?X) :- q(?X).
?-p(?X).

I can get the p('a') as a result.
but if I know p(?X) :- q(?X). Then I want to get the ¬q(?X) :- ¬p(?X). How can I realize this?

Currently we are rate limited here. So, wait a bit before merging anything. Let's try restarting in ~20min of this issue being opened.

Getting this message when trying to create feedstocks here. Here is the traceback in the log. Almost looks like rate limiting, but not quite.

Can we establish a standard for how we do the whole python setup.py dance?

Please vote by posting a comment with the corresponding number. Majority after 1 week wins?

1. python setup.py install --single-version-externally-managed --record=record.txt
2. python setup.py install --single-version-externally-managed --root=/
3. python setup.py install --old-and-unmanageable
4. python -m pip install --no-deps .
5. pip install --no-deps .

I give up on strikethrough. 4 is out, due to error (https://travis-ci.org/conda-forge/staged-recipes/jobs/127374916#L405)

Each time a PR is submitted, example is built. We really don't need to do that.
We could just remove the recipe before conda-build-all in the CI scripts (e.g. https://github.com/conda-forge/staged-recipes/blob/master/appveyor.yml#L72).

@pelson I want to start packaging geos and shapely in the conda-forge channel.

Should I create a feedstock repo in the conda-forge organization or should I send the recipes here to staged-recipes?

Hey folks,

Something I said in the comments of the merged RIOS PR:

I'm not convinced you should relax your gdal 2.0.2 spec (or perhaps it should become a >= spec). The TravisCI output suggests that rios isn't working with gdal 2.0.0. You need to prevent people from installing this package alongside 2.0.0, then.

I wanted to make sure that this doesn't get lost in an already closed/merged thread, because it doesn't just apply to RIOS, it applies across the entire conda-forge ecosystem.

It is important that a package's run requirements have sufficient version requirements to ensure proper operation. In particular: it is not sufficient to assume that a user will have, or obtain, the latest version of a given dependency. If you know your package is not compatible with older versions of a dependency, you must reflect that in the run dependencies.

For example: suppose I have a functioning environment built containing gdal 2.0.0 from defaults; then I type conda install rios. If rios simply specifies gdal as a run-time dependency, with no version specification, then it will happily install it without upgrading gdal. But rios will be broken according to this. This means that an unqualified runtime dependency on gdal is incorrect. On the other hand, if rios includes gdal >2.0 as its runtime dependency, it will cause conda to upgrade past the failure.

On the other hand, you want your dependencies to be as relaxed as possible, otherwise you invite a litany of unsatisfiable specs in complex environments. So, for example, the current rios recipe specifies gdal 2.0.2. In my view, this ought to be relaxed to gdal >2.0 or gdal >=2.0.2, unless you have good reason to believe that subsequent versions of gdal will break rios.

To be fair, the problems here are made worse by channel collision issues: it could be that conda-forge's version of gdal 2.0* works just fine, but the ones in default do not. We're working on that, as you know. But for the foreseeable future, people are going to be mixing and matching packages from defaults and conda-forge, so one should still take care to lock down dependencies to avoid breakage.

See this comment from @scopatz. Basically, reviewers and contributors could hopefully do less work if we have a few common examples to look at that demonstrate ideal usage patterns.

For instance, we are commonly pointing out that we want to use script for one line statements. We should have this in the example. Also, setuptools arguments should be included. We might even want to explain they should be dropped for distutils only installs.

Having an example of the UNIX only recipe would be very helpful for showing how to skip Windows.

Also, having a cross platform (or at least Windows) build so that we can demonstrate features.

These should be simple to come up with, but are a huge benefit as they reward users for simply copying and tweaking the examples.

We don't upload anything in staged-recipes. Maybe we leave the conda-forge account for the feedstocks and use another one here? Does it makes sense?

I'm already feeling overwhelmed with keeping track of recipes that either need Windows modification or need attention at some later date (e.g. to include some Continuum changes that I know are available)

Is there any way we can consolidate these recipes/repos into a list that is easier to approach and to track? Perhaps a wiki page? Some Google spreadsheet? Create issues on those repos and assign people to them?

Just a thought on refining the jobs in the CI build queues when dealing with PRs. Frequently people are tweaking their PRs based on the quick feedback from the community (thanks for being so great everyone 😄). Unfortunately, this results in CI backlog of builds particularly for the slower ones (namely AppVeyor, sometimes Travis, very rarely Circle). It would be good if we just "pushed" the cancel button. I do this manually to some extent, but it is hard to keep up with and not always a good use of time. It is not obvious to me where this would run. Though putting it on the fastest of the CIs would make sense (so Circle). Thoughts? Other ideas?

EDIT: This has moved to these docs in this PR ( conda-forge/conda-forge.github.io#95 ). So I have cut it from here to avoid unnecessary duplication and to avoid this potentially becoming stale/contradictory. However, it is still important to list people's general preferences regarding what they are maintaining generally to avoid overpinging. This issues has been restructured accordingly.

The packages used to be under ../../packages/source/<first letter>/<pkg name>/, now only available as ../../packages/<two-char hash>/<two-char hash>/<long hash>/.
The according links also changed in the "simple" index page. This has changed the disk layout of the bandersnatch. Our build system broke due to the mirror of bandersnatch doe not have the same disk layouts in the disk.

Related issues:

• https://sourceforge.net/p/pypi/support-requests/626/
• https://bitbucket.org/pypa/pypi/issues/447/direct-links-of-packages-gone

Example:

There is a package colorlog with two versions: 2.6.1 (old) and 2.6.3 (new). The following link works fine:

https://pypi.python.org/packages/source/c/colorlog/colorlog-2.6.1.tar.gz

However, if you just change the version number, you will get 404:

https://pypi.python.org/packages/source/c/colorlog/colorlog-2.6.3.tar.gz

The working link to 2.6.3 version is:

https://pypi.python.org/packages/cd/e7/7e14ce72038e83bc475a1f708485e5ea4a789beef26ff64b732918f860e6/colorlog-2.6.3.tar.gz

If someone tries to add a recipe for a feedstock that is already present, it would be nice if the differences could be placed in a PR against the existing feedstock. Ideally, the PR would ping the user who made the change and, if possible, make them the author of the PR so they could change it as needed.

It seems nbconvert uses inkscape to transform svg images to whatever is used to include such images in latex/PDF files.

There is a recipe in conda-recipes: https://github.com/conda/conda-recipes/tree/master/inkscape but this uses a repackaged windows download.

Not sure how hard it would be to use a build from source approach... It uses MSYS/MINGW: http://wiki.inkscape.org/wiki/index.php/Compiling_Inkscape_on_Windows and especially the devlibs repo http://wiki.inkscape.org/wiki/index.php/Inkscape_Devlibs makes me too scared to try... :-(

Some recipes utilise intrinsics such as SSE and AVX. Enabling these intrinsics greatly improves the speed of many applications. However, most applications do not guard against a particular intrinsic not existing at run time and will simply crash if the architecture it is run on does not support a given instruction.

For example, OpenCV supports up to SSE 4.1 instructions - yet it is feasible that someone may download OpenCV on an older Linux box that does not support SSE 4.1. Therefore, I imagine we want to do something like the proposal in #80 whereby we have features for different SSE levels - where supported?

I'm wondering what the best way of packaging Sphinx-contrib extension would be. I'm actually using only sphinxcontrib-mockautodoc, and I have its recipe in our company's recipes repo.

Hi there,

I'm running into one dependency after the other.

Most of the time I Google for "homebrew formular packagexyz"
and copy parts from there.

Why not write a wrapper for those formulas?

AppVeyor badge appears to be broken. @pelson is staged-recipes still in your account? I cannot see the master logs either.

Travis-CI failed during the merge of #100 and I had to re-start it. I Wonder if we should send the status e-mail to the mailing list, or to a few project managers.

It would be nice if we checked if skip: True was set for a platform as early as possible and terminate CI.

Seems like this might not be necessary (unless I am missing something). Skipping running these builds would shorten our build queue on all CIs if we fast finish them.

Repository registered at github, now call 'conda smithy register-ci'
Making feedstock for poliastro
/Users/travis/build/conda-forge/staged-recipes/recipes/poliastro has some lint:
  Selectors are suggested to take a "  # [<selector>]" form.
Traceback (most recent call last):
  File ".CI/create_feedstocks.py", line 93, in <module>
    subprocess.check_call(['conda', 'smithy', 'recipe-lint', recipe_dir])
  File "/Users/travis/miniconda/lib/python3.5/subprocess.py", line 584, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda', 'smithy', 'recipe-lint', '/Users/travis/build/conda-forge/staged-recipes/recipes/poliastro']' returned non-zero exit status 1

I am working on that.

Unless I am mistaken, this step merely lints the recipe. It would be good if we just change this build to run on the Linux container infrastructure. The reason being this will start a lot faster and thus give users feedback more quickly on whether the recipe is acceptable.

We still need to set the CONDA_NPY dummy environment on Linux. I guess that is because we are still an older conda-build (1.18.2) there.

Traceback (most recent call last):
  File "/opt/conda/bin/conda-build-all", line 9, in <module>
    load_entry_point('conda-build-all==0.11.0', 'console_scripts', 'conda-build-all')()
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/cli.py", line 85, in main
    b.main()
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 207, in main
    recipe_metas = self.fetch_all_metas()
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 150, in fetch_all_metas
    recipe_metas = sort_dependency_order(recipe_metas)
  File "/opt/conda/lib/python3.5/site-packages/conda_build_all/builder.py", line 97, in sort_dependency_order
    meta.parse_again()
  File "/opt/conda/lib/python3.5/site-packages/conda_build/metadata.py", line 356, in parse_again
    self.meta = parse(self._get_contents(permit_undefined_jinja))
  File "/opt/conda/lib/python3.5/site-packages/conda_build/metadata.py", line 634, in _get_contents
    env.globals.update(context_processor(self, path))
  File "/opt/conda/lib/python3.5/site-packages/conda_build/jinja_context.py", line 65, in context_processor
    ctx = get_environ(m=initial_metadata)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/environ.py", line 248, in get_dict
    d['PKG_BUILD_STRING'] = str(m.build_id())
  File "/opt/conda/lib/python3.5/site-packages/conda_build/metadata.py", line 461, in build_id
    for ms in self.ms_depends():
  File "/opt/conda/lib/python3.5/site-packages/conda_build/metadata.py", line 434, in ms_depends
    ms = handle_config_version(ms, ver)
  File "/opt/conda/lib/python3.5/site-packages/conda_build/metadata.py", line 307, in handle_config_version
    raise RuntimeError("'%s' requires external setting" % ms.spec)
RuntimeError: 'numpy x.x' requires external setting

See https://circleci.com/gh/conda-forge/staged-recipes/2056 and #381 (comment)

Ping @pelson

Related: #395
Related: #460

See this line in the log. For some reason, it is not able to setup the CircleCI queue for the feedstock.

Possibly a collision here. Restarting one of them.

• https://travis-ci.org/conda-forge/staged-recipes/jobs/117766307#L258
• https://travis-ci.org/conda-forge/staged-recipes/jobs/117767354#L308

Currently we are rate limited here. So, wait a bit before merging anything. Let's try restarting in ~20min of this issue being opened.

The usecase would be that one could install pandoc (already packaged) and latex to get pdf output form nbconvert. For that to work it would be nessesary if the tex distribution would install the binaries into <env>\Scripts on windows.

Unfortunately, this seems to be a lot harder than I thought (only tried windows):

repackage miktex portable

miktex directory structure means that the bin dir has to be separate from the script dir in the conda env (bin is in .\miktex\bin and it expects the rest of the stuff in ., so we end up with a directory structure below the conda env main dir :-/). But this means that we have to add .\miktex\bin to the path, which is feasible using activate.d scripts, but this again means that one would have to activate such an environment before starting the notebook server (or any other commandline program), which is IMO not gonna work (e.g. who activates the main environment? And my notebook server is installed in an env, but is started by just specifying the complete path to jupyter.exe in that env).

Miktex has the disadvantage that it is only available on windows, but the advantage, that the install size is little because it installs (can install) missing packages automatically.

texlive

The portable binary installer installs itself into ./bin/win32, so that's also not gonna work :-( It seems it is possible to install texlive from source in the required form (setting bindir in the configure step), but just reading the documentation made me too scared to try (and it uses svn, which I don't have installed...). And actually supporting a latex distributions is a bigger job than I want to try...

texlife is available in mac/linux/windows, but installing missing packages is a manual step (or the package gets really big...)...

So, what are the options?

• Use a repackaged miktex portable, use a activate.d script and warn user, that they have to use activate (I've actually no clue how that work out in the root environment), if they want to have access to the latex binaries in e.g. nbconvert. Maybe get nbconvert, mpl and friends to add the dir if it exist (either upstream or in a patch).
• Try to compile texlive on all platforms, putting the bindir in the dir which is anyway included in the path
• I've no clue if it would be possible to edit the binaries of texlive to remove the \win32 part?

Any (other) ideas?