Move the build script to intake-esm about intake-esm-datastore HOT 9 CLOSED

intake_esm already supports building collections out of the box:

col = open_esm_metadatastore(collection_input_definition)

The advantage of this script is that you can build collections in the background; I have a SLURM batch script, for instance, that wraps build_collection.py...and would like to check this in too.

I think I like having the capacity live here because I can update the repo and have a ready tool to update the collections I am using.

from intake-esm-datastore.

I'm thinking of something different..something along the lines of having setuptools copy the script to PATH and make it available for general use with:

build_collection collection-definitions/my-collection.yml --overwrite-existing

from intake-esm-datastore.

Ok, that works. Where on PATH?

from intake-esm-datastore.

Where on PATH?

In the same environment as intake-esm. For instance for conda, it will put it in the bin directory of the environment:

root@0a9165cd7ff7:/usr/local/envs/intake-esm-dev/bin# which intake-esm-builder 
/usr/local/envs/intake-esm-dev/bin/intake-esm-builder

from intake-esm-datastore.

Is there any way to ensure that the script is executed in the right environment? We probably can't assume conda, but it would be great if the shebang string could actually set the conda environment.

from intake-esm-datastore.

Is there any way to ensure that the script is executed in the right environment?

setuptools takes care of everything: https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

and because of #! /usr/bin/env python shebang, the python used will correspond to whichever python environment is active when the script is invoked.

from intake-esm-datastore.

But I think that if a user runs the script from an environment that doesn't have intake_esm installed, the script will fail.

from intake-esm-datastore.

The intake-esm-builder command is exclusively available in an environment with intake-esm installed.

abanihi@casper01: /glade/work/abanihi/devel/ncar $ which intake-esm-builder
/glade/work/abanihi/softwares/miniconda3/envs/dev/bin/intake-esm-builder
abanihi@casper01: /glade/work/abanihi/devel/ncar $ which python
/glade/work/abanihi/softwares/miniconda3/envs/dev/bin/python

abanihi@casper01: /glade/work/abanihi/devel/ncar $ conda activate base
abanihi@casper01: /glade/work/abanihi/devel/ncar $ intake-esm-builder
bash: intake-esm-builder: command not found

As a result, if a user tries running the script from an environment that doesn't have intake_esm installed, the script will fail and this is what we want, right?

from intake-esm-datastore.

I just wish there was a way to designate the right environment within the script itself; the shebang string tells the shell with program to use...it doesn't seem completely inconceivable that it also communicates the environment.

from intake-esm-datastore.