rapidsai / gpuci-tools Goto Github PK

View Code? Open in Web Editor NEW

1.0 2.0 10.0 91 KB

User tools for use within the gpuCI environment

License: Apache License 2.0

Shell 100.00%

gpuci-tools's Introduction

gpuci-tools

User tools for use within the gpuCI environment

Install gpuCI tools

`wget` method

# as root
wget https://github.com/rapidsai/gpuci-tools/releases/latest/download/tools.tar.gz -O - | tar -xz -C /usr/local/bin

# as non-root
wget https://github.com/rapidsai/gpuci-tools/releases/latest/download/tools.tar.gz -O - | sudo tar -xz -C /usr/local/bin

`conda` method w/o mamba

conda install -c gpuci gpuci-tools

`conda` method w/ mamba

conda install -c gpuci -c conda-forge gpuci-tools mamba

`curl` method

curl -s https://raw.githubusercontent.com/rapidsai/gpuci-tools/main/install.sh | bash
source ~/.bashrc
cd ~

`git` method

git clone https://github.com/rapidsai/gpuci-tools.git .gpuci
chmod +x .gpuci/tools/*
export PATH="$PWD/.gpuci/tools:$PATH"

gpuci-tools's People

Contributors

Stargazers

Watchers

Forkers

aleksficek sean-frye rlratzel dillon-cullinan mike-wendt raydouglass mdemoret-nv bdice madsbk ayodeawe

gpuci-tools's Issues

Remove extraneous package types / install methods

This repository is just a collection of shell scripts. It has no external dependencies and can be installed simply with one of the wget ... | tar ... commands below:

# as root
wget https://github.com/rapidsai/gpuci-tools/releases/latest/download/tools.tar.gz -O - | tar -xz -C /usr/local/bin

# as non-root
wget https://github.com/rapidsai/gpuci-tools/releases/latest/download/tools.tar.gz -O - | sudo tar -xz -C /usr/local/bin

# or
wget https://github.com/rapidsai/gpuci-tools/releases/latest/download/tools.tar.gz -O - | tar -xz -C ~/.local/bin

Therefore, having other package types and install scripts, like a conda package and install.sh, just represents unnecessary overhead for us to maintain. Especially the conda package, which requires us to manage a job with a set of credentials to upload to Anaconda.org.

With #27 merged, every set of changes is automatically uploaded to a new GitHub release.

With all that being said, I think we should remove the conda package, the install.sh script, and update the README.md to reflect these changes in order to keep this repository as simple as possible.

[FEA] Update `gpuci_conda_retry` to not break if `conda` is supplied

To use gpuci_conda_retry it does not follow the same convention as gpuci_retry. That is you currently need to drop conda from you command to use it.

Given this command:

$ conda install -y -c gpuci gpuci-tools

To use it with gpuci_retry we simply add it to the beginning of the command:

$ gpuci_retry conda install -y -c gpuci gpuci-tools

However, doing the same with gpuci_conda_retry results in an error:

$ gpuci_conda_retry conda install -y -c gpuci gpuci-tools
CommandNotFoundError: No command 'conda conda'.

Idea here is to update the script so if the above is passed, gpuci_conda_retry will not add conda to the command.

JSONDecodeErrors not being retried

It seems that PR #8 isn't working for all JSONDecodeErrors.

Job Info

Job Link

Job failure output:

Step 29/61 : RUN gpuci_conda_retry install -y -n rapids         "rapids-notebook-env=${RAPIDS_VER}*"     && conda remove -y -n rapids --force-remove         "rapids-notebook-env=${RAPIDS_VER}*"
 ---> Running in 5953d3b479ff
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - rapidsai-nightly/linux-64::ucx==1.8.1+g6b29558=cuda11.0_0
  - rapidsai-nightly/linux-64::ucx-py==0.16.0a201015=py37_g6b29558_187
done

## Package Plan ##

  environment location: /opt/conda/envs/rapids

  added / updated specs:
    - rapids-notebook-env=0.16


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    rapids-notebook-env-0.16.0a201016|cuda11.0_py37_ga8d4495_78           8 KB  rapidsai-nightly
    ------------------------------------------------------------
                                           Total:           8 KB

The following NEW packages will be INSTALLED:

  rapids-notebook-e~ rapidsai-nightly/linux-64::rapids-notebook-env-0.16.0a201016-cuda11.0_py37_ga8d4495_78



Downloading and Extracting Packages

rapids-notebook-env- | 8 KB      |            |   0% 
rapids-notebook-env- | 8 KB      | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.8.5

Please update conda by running

    $ conda update -n base -c defaults conda



# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "/opt/conda/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
        return getattr(module, func_name)(args, parser)
      File "/opt/conda/lib/python3.8/site-packages/conda/cli/main_remove.py", line 86, in execute
        txn = solver.solve_for_transaction()
      File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 123, in solve_for_transaction
        self._notify_conda_outdated(link_precs)
      File "/opt/conda/lib/python3.8/site-packages/conda/core/solve.py", line 960, in _notify_conda_outdated
        SubdirData.query_all(conda_newer_spec, self.channels, self.subdirs,
      File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 105, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
        yield fs.pop().result()
      File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 439, in result
        return self.__get_result()
      File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
        raise self._exception
      File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 97, in <lambda>
        subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
      File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 110, in query
        self.load()
      File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 174, in load
        _internal_state = self._load()
      File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 276, in _load
        _internal_state = self._process_raw_repodata_str(raw_repodata_str)
      File "/opt/conda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 349, in _process_raw_repodata_str
        json_obj = json.loads(raw_repodata_str or '{}')
      File "/opt/conda/lib/python3.8/json/__init__.py", line 357, in loads
        return _default_decoder.decode(s)
      File "/opt/conda/lib/python3.8/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/opt/conda/lib/python3.8/json/decoder.py", line 353, in raw_decode
        obj, end = self.scan_once(s, idx)
    json.decoder.JSONDecodeError: Unterminated string starting at: line 390529 column 7 (char 11238796)

`$ /opt/conda/bin/conda remove -y -n rapids --force-remove rapids-notebook-env=0.16*`

  environment variables:
                 CIO_TEST=<not set>
                CONDA_DIR=/opt/conda
               CONDA_ROOT=/opt/conda
          LD_LIBRARY_PATH=/usr/local/gcc7/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/u
                          sr/local/cuda/lib64:/usr/local/lib
             LIBRARY_PATH=/usr/local/cuda/lib64/stubs
                     PATH=/usr/local/gcc7/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cu
                          da/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
               PYTHON_VER=3.7
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : None
       user config file : /root/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 4.8.3
    conda-build version : not installed
         python version : 3.8.3.final.0
       virtual packages : __glibc=2.17
       base environment : /opt/conda  (writable)
           channel URLs : https://conda.anaconda.org/gpuci/linux-64
                          https://conda.anaconda.org/gpuci/noarch
                          https://conda.anaconda.org/rapidsai-nightly/linux-64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/rapidsai/linux-64
                          https://conda.anaconda.org/rapidsai/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.23.0 CPython/3.8.3 Linux/5.4.0-1028-aws centos/7.8.2003 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

Upload successful.
The command '/bin/bash -c gpuci_conda_retry install -y -n rapids         "rapids-notebook-env=${RAPIDS_VER}*"     && conda remove -y -n rapids --force-remove         "rapids-notebook-env=${RAPIDS_VER}*"' returned a non-zero code: 1
Build step 'Execute shell' marked build as failure
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[WS-CLEANUP] done
Finished: FAILURE

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.