paperspace / gradient-cli Goto Github PK

View Code? Open in Web Editor NEW

65.0 16.0 23.0 3.5 MB

The command line interface for Gradient - https://gradient.paperspace.com

License: ISC License

Makefile 0.13% Python 99.42% Jupyter Notebook 0.38% Dockerfile 0.07%

gradient paperspace machine-learning machine-learning-library python ai deep-learning

gradient-cli's Introduction

Note We are rolling out a new streamlined Paperspace CLI and recommend using this new CLI for all new projects.

Gradient CLI

Get started: Create Account • Install CLI • Tutorials • Docs

Resources: Website • Blog • Support • Contact Sales

Gradient is an an end-to-end MLOps platform that enables individuals and organizations to quickly develop, train, and deploy Deep Learning models. The Gradient software stack runs on any infrastructure e.g. AWS, GCP, on-premise and low-cost Paperspace GPUs. Leverage automatic versioning, distributed training, built-in graphs & metrics, hyperparameter search, GradientCI, 1-click Jupyter Notebooks, our Python SDK, and more.

Key components:

Notebooks: 1-click Jupyter Notebooks.
Workflows: Train models at scale with composable actions.
Inference: Deploy models as API endpoints.

Gradient supports any ML/DL framework (TensorFlow, PyTorch, XGBoost, etc).

See releasenotes.md for details on the current release, as well as release history.

Getting Started

Make sure you have a Paperspace account set up. Go to http://paperspace.com to register and generate an API key.
Use pip, pipenv, or conda to install the gradient package, e.g.:

pip install -U gradient

To install/update prerelease (Alpha/Beta) version version of gradient, use:

pip install -U --pre gradient
Set your api key by executing the following:

gradient apiKey <your-api-key-here>

Note: your api key is cached in ~/.paperspace/config.json

You can remove your cached api key by executing:

gradient logout

Executing tasks on Gradient

The Gradient CLI follows a standard [command] [--options] syntax

For example, to create a new Workflow in a project use:

gradient projects list
gradient workflows create --name <name> --projectId <project-id>

For a full list of available commands run gradient workflows --help. You can also view more info about Workflows in the docs.

Contributing

Want to contribute? Contact us at [email protected]

Pre-Release Testing

Have a Paperspace QA tester install your change directly from the branch to test it. They can do it with pip install git+https://github.com/Paperspace/gradient-cli.git@MYBRANCH.

gradient-cli's People

Contributors

Stargazers

Watchers

gradient-cli's Issues

Some parameters in job_client.create() don't work

I'm following the sample docs at gradient.api_sdk.clients.job_client. I can't get working_directory or job_env to work.

working_directory: auto set to /paperspace. When I pass in /app, it's ignored & /paperspace is still used (listed Console > Job > Environment; and on the Job model fetch from job_client.list()). I just cashed in and set all my Docker stuff from /app to /paperspace and it worked. Ideally I wouldn't even want to pass /app, but it'd just be picked up from WORKDIR in the Dockerfile (which is /app)

job_env: I can't find a workaround (save putting a config.json in /storage). Parameter gets gets ignored; of note, the docs has a typo:

job = job_client.create(
    ...
    job_env={
        'CUSTOM_ENV'='Some value that will be set as system environment',
    }
)

'CUSTOM_ENV'='..' should be 'CUSTOM_ENV': '..'. Not nit-picking; just that it had me second guessing my approach, since using a dict gets ignored ([j.job_env for j in job_client.list()] => [None, None, ..]) so I tried json.dumps(my_dict), still ignored. Any suggestions on getting the job_env passed in?

Overall, it seems like some params are respected, and others not; and it's hard to know which is which. Maybe a mismatch on the docs vs Github vs PyPi?

'gradient' is not recognized as an internal or external command, operable program or batch file on Windows 11

I'm trying to get access to gradient cli and getting the following error: 'gradient' is not recognized as an internal or external command,
operable program or batch file on Windows 11. My understanding is it has something to do with environment variables. What would be a quick fix for this?

PyYAML 5.4.1 is broken, use PyYAML 5.3.1 instead

PyYAML version 5.4.1 is causing 'pip install gradient' to fail.

See: yaml/pyyaml#724

Solution is to use 5.3.1

ie., change line 70 in setup.py from PyYAML==5.* to PyYAML==5.3.*

Collecting PyYAML==5.* (from gradient)
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      /private/var/folders/fd/b8flfqkj3_bf4hp3dnr1l2hh0000gn/T/pip-build-env-usxd2f4p/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

Failed to create resource

running the command:
gradient notebooks start --id [id] --machineType Free-CPU

gives an output
Failed to create resource: We are currently out of capacity for the selected VM type. Try again in a few minutes, or select a different instance.

But, if I tried on Web UI. it works. I think it doesn't synchronize really well

Request to Increase Supported Marshmallow Version

Hello,

I am currently using the gradient package in a project that also depends on the langchain package. Unfortunately, I've run into a dependency conflict between these two packages, specifically with regards to the marshmallow library.

gradient currently supports marshmallow versions below 3.0, while langchain (via dataclasses-json) requires marshmallow version 3.3.0 or later. Here are the specific version requirements:

gradient: requires marshmallow versions <3.0
langchain: requires dataclasses-json version >=0.5.7,<0.6.0
dataclasses-json: requires marshmallow version >=3.3.0,<4.0.0

I wanted to ask if there are any plans to update gradient to support newer versions of marshmallow, or if there is a specific reason why gradient requires marshmallow version below 3.0?

It would be a great help if gradient could be updated to support later versions of marshmallow to resolve this dependency conflict.
If updating gradient to support newer versions of marshmallow is not possible or would be too disruptive, I would appreciate any advice or recommendations how to solve it on my end! Thanks!

Failed to execute request against storage provider when uploading a dataset

Dear everyone,

I am trying to upload a new version of my dataset, but I keep getting Brokenpipe errors.
Here is what I am doing :

gradient datasets versions create --id ...
gradient datasets files put --id ...:... --source-path "."

Failed to execute request against storage provider: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

Uploading to other cloud providers is working, but not here.

By the way, is gradient-cli uploading everything again at each new version? Or is it comparing hashes to avoid unnecessary file transfers?

Thanks,
Clément

PyYAML dependency issue

Description
I am integrating gradient in my Python backend but cannot because of gradient requirements concerning PyYAML.

gradient 2.0.6 depends on PyYAML==5.*
All my dependencies require PyYAML>=6.*. Can you please upgrade or let me know how to do it?

Why there is no deployments interfaces in SDK, only can be found in CLI?

I want to use the SDK to operate deployments, but I found that the

deployment_client = sdk_client.DeploymentsClient()

listed in https://docs.paperspace.com/gradient/python-sdk/sdk-tutorial/ not work. There is also no corresponding file in the code. Has it been removed?

Problem running gradient in the cli for Win10

This is more to me documenting a simple problem when running it in Windows 10.

I tried installing gradient into my windows 10, but encountered an error when running it. I tried with virtualenv and venv, both had the same issue. I'm using python 3.8. Pip version is 21.0.1. I have tried deleting the environment, starting over, tried skipping the cache also.

This is the error screenshot

A bit of google-ing, it seems I had to separately install a windows version of curses. Installing this fixed the problem https://pypi.org/project/windows-curses/

Support gradient_statsd from the CLI

Hi,

Is it possible to specify the metric names when creating a job (like the Job Builder)? There doesn't appear to have a parameter for that in the gradient jobs create params list.

Thanks!

my ip is banned by cloudflare

Error 1008 Ray ID: 7c2032b55e57df8f • 2023-05-04 10:53:06 UTC
Access denied
What happened?

The owner of this website (api.paperspace.io) has banned your IP address (182.253.163.132).

Cloudflare Ray ID: 7c2032b55e57df8f

Gradient CLI won't list available machine types

Hello. The following command:
gradient clusters machineTypes list
never returns anything besides No data found
API key is set correctly, other commands do work.

What could possibly be the issue?

CLI Cannot find Job Artifacts

Browsing through the Web console I can see artifacts listed for the job, but when doing gradient jobs artifacts list <jobid> it always tells me "no data found"

Failing to start notebook

I'm trying to start an instance of an existing notebook:
gradient notebooks start --id [id] --machineType Free-P5000

and get in return:
Failed to create resource: Cluster null not found

specifying any cluster ID doesn't change anything and the output remains the same.

A bug ?

dead link in documentation

Dear Gradient-CLI Team,
I'm not sure if this is the right place to report this, but I noticed a link in the documentation is not working.
On page:
https://docs.paperspace.com/gradient/projects/gradientci-v2

there is a link to the config documentation:
https://github.com/Paperspace/Docs/tree/dace77bbd70b824f1dfd4bf27fc24deef5218d16/experiments/gradient-config.yaml.md

Unfortunateley I get a 404 following it.
Greetings
Orys

Access artifacts across jobs?

Is it possible to access an artifact from one job inside another job?

can not instantiate ipywidgets.Output in notebook

not sure if it is the right place but our users encounters the following errors when using DiscoArt on paperspace

from ipywidgets import Output
Output()

Could you please give it a check

ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly

I'm having an issue installing gradient with pip in anaconda. Is this supported?
pip install -U gradient

ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly

System is macOS Big sur 11.3.1
Python 3.9
Anaconda 4.9.2
pip 21.1.1

pip install -U gradient --no-use-pep517 fails too, but its not something I want to focus on.

Running any gradient command produces TypeError

Running any gradient command in the terminal produces the same error. Whether I run gradient by itself (which should produce a help message) or if I try to set my API key, the same error gets produced. Tracebacks are shown below:

~$ gradient
Traceback (most recent call last):
File "/home/kevin/.local/bin/gradient", line 5, in
from gradient import main
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/init.py", line 1, in
from .api_sdk import *
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/init.py", line 1, in
from .clients import *
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/clients/init.py", line 1, in
from .clusters import ClustersClient
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/clients/clusters.py", line 1, in
from gradient.api_sdk import repositories
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/repositories/init.py", line 1, in
from .clusters import ListClusters
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/repositories/clusters.py", line 7, in
from gradient.api_sdk.serializers.cluster import ClusterSchema
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/serializers/init.py", line 1, in
from .artifact import ArtifactSchema
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/serializers/artifact.py", line 3, in
from gradient.api_sdk import models
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/models/init.py", line 4, in
from .deployment import Deployment
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/models/deployment.py", line 5, in
class Deployment(object):
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/models/deployment.py", line 76, in Deployment
tags = attr.ib(type=list, factory=list)
TypeError: attrib() got an unexpected keyword argument 'factory'

~$ gradient version
Traceback (most recent call last):
File "/home/kevin/.local/bin/gradient", line 5, in
from gradient import main
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/init.py", line 1, in
from .api_sdk import *
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/init.py", line 1, in
from .clients import *
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/clients/init.py", line 1, in
from .clusters import ClustersClient
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/clients/clusters.py", line 1, in
from gradient.api_sdk import repositories
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/repositories/init.py", line 1, in
from .clusters import ListClusters
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/repositories/clusters.py", line 7, in
from gradient.api_sdk.serializers.cluster import ClusterSchema
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/serializers/init.py", line 1, in
from .artifact import ArtifactSchema
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/serializers/artifact.py", line 3, in
from gradient.api_sdk import models
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/models/init.py", line 4, in
from .deployment import Deployment
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/models/deployment.py", line 5, in
class Deployment(object):
File "/home/kevin/.local/lib/python3.6/site-packages/gradient/api_sdk/models/deployment.py", line 76, in Deployment
tags = attr.ib(type=list, factory=list)
TypeError: attrib() got an unexpected keyword argument 'factory'

Update older example or create new ML5js example PLEASE

It's fun to learn through tutorials but they should work.

I'm following this tutorial. And it's outdated and should be updated because it's a very easy way of learning with light tools.

https://blog.paperspace.com/creating-your-own-style-transfer-mirror/

I get this error, when trying to overcome all the outdated obstacles.
But now I'm stuck without a proper log.

gradient experiments create singlenode --container cvalenzuelab/styletransfer --machineType P5000 --command './run.sh' --projectId 'Style Transfer training'
Failed to create resource: Project not found for handle Style Transfer training
Project not found

Please help there are more people on the community page that want this to work. But got no reaction from Paperspace

Sorry for the begging :)

Update version specifier

gradient-cli/setup.py

Line 70 in 9eb645e

'PyYAML==5.*',

is no longer a valid version specifier.

error in gradient setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected end or semicolon (after version specifier)
    PyYAML>=5.*
          ~~~^

Affects also websocket-client==0.57.*

Support for click < 8

At lot of distribution are shipping click > 8. It would be nice to get support for click > 8 in the near future. Currently gradient is failing to build or to install on various distributions (like NIx https://hydra.nixos.org/build/153576613) as the requirement is pinned ( click==7.1.2).

Thanks a lot.

"gradient experiments list" fails

Running gradient experiments list in the CLI fails with this stack trace:

Traceback (most recent call last):
  File "~/miniconda3/envs/gradient/bin/gradient", line 8, in <module>
    sys.exit(main.main())
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/main.py", line 7, in main
    _cli_entry_point()
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/cli/cli.py", line 17, in main
    super(GradientGroup, self).main(*args, **kwargs)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/cli/experiments.py", line 701, in list_experiments
    for experiments_str, next_iteration in res:
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/commands/common.py", line 73, in _generate_data_table
    **kwargs
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/commands/experiments.py", line 243, in _get_instances
    instances, meta_data = self.client.list(get_meta=True, **kwargs)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/api_sdk/clients/experiment_client.py", line 741, in list
    experiments = repository.list(project_id=project_id, limit=limit, offset=offset, get_meta=get_meta, tags=tags)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/api_sdk/repositories/common.py", line 97, in list
    instances = self._get_instances(response, **kwargs)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/api_sdk/repositories/common.py", line 108, in _get_instances
    objects = self._parse_objects(response.data, **kwargs)
  File "~/miniconda3/envs/gradient/lib/python3.6/site-packages/gradient/api_sdk/repositories/experiments.py", line 45, in _parse_objects
    experiment_dict.update(experiment_dict["templateHistory"]["params"])
KeyError: 'params'

Using version 0.6.0

Installation Suggestion: use pipx gradient

You might want to suggest using pipx to install the gradient CLI:
https://github.com/pipxproject/pipx

The nice thing about pipx is that it:

doesn't require putting gradient and required libraries in the system or user base python installations
would put the gradient binary in the user's path no matter which virtual env the user is using
makes its own virtual environment just for the gradient tool and handles all of that
works mostly like pip, except with the above benefits.

Conda package version of gradient-cli

Would Paperspace prefer to be responsible for maintaining a conda package, or would it be okay with community-led packaging? In either case, my preferred channel would be conda-forge.

When providing --ignoreFiles comma separated the code returns Attribute error.

The command below with the --ignoreFiles field set

gradient experiments run single node \
--name test \
--projectid 12345 \
--container paperspace/tensorflow-python \
--machineType V100 \
--command 'python main.py' \
--ignoreFiles "file1,file2,file3,folder1,folder2,folder3"

Returns the following error

file_paths = self._retrieve_file_paths(workspace_path, ignore_files)
File "/lib/python3.7/site-packages/gradient/workspace.py", line 59, in _retrieve_file_paths
    exclude += ignored_files.split(',')
AttributeError: 'list' object has no attribute 'split'

Keep artifacts for canceled jobs

I am aware this is probably not the place to open this issue, but it seems like artifacts are not kept when a job is canceled. But why? I just finished a 2 days running job and wanted to have access to my best model, but the artifacts are not showing up potentially because I canceled the job. If this is the case, this is very frustrating.

Unable to install due to numpy==1.19.4

Hi I am unable to use gradient on python 3.7.9 due to RuntimeError caused by numpy==1.19.4

RuntimeError: The current Numpy installation ('c:\users\XXXX\envs\gradient\lib\site-packages\numpy\init.py') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86

I tried downgrading the numpy, but it appears to be incompatible and i got the ff ModuleNotFoundError:
ModuleNotFoundError: No module named '_curses'

Update Readme.md

I found two outdated or currently not working commands while trying to execute my container:

The --project option was ignored but providing --projectId worked.
The example for the --command option had as error message:
Error: Missing argument "SCRIPT...". Upon providing a script the originally provided command was ignored. Instead I had to use the --shell option to execute the correct call.

Experiment worker command base64-encoded

gradient experiments details --id <id> outputs a table with various data. The "Worker Command" row is presented encoded in base64. Steps to reproduce should just be running that command with some experiment id. Feel free to ask if you need any more detail from my particular experiment.