Giter VIP home page Giter VIP logo

prior's Introduction

🐍 PRIOR

A Python Package for Seamless Data Distribution in AI Workflows

DALL·E 2022-09-12 18 02 32 - A friendly green snake typing on a computer on the floor

Installation

Install the prior package with pip:

pip install prior

Datasets

import prior
prior.load_dataset("procthor-10k")
import prior
prior.load_dataset("object-nav-eval")

Models

import prior
prior.load_model(project="procthor-models", model="object-nav-pretraining")

Example Usage

To use a public Python dataset, simply run:

import prior
dataset = prior.load_dataset("test-dataset", entity="mattdeitke", revision="main")

Here, revision can be either a tag, branch, or commit hash.

Private Datasets

If you want to use a private dataset, make sure you're either:

  1. Already logged into GitHub from the command line, and able to pull a private repo.
  2. Set the GITHUB_TOKEN environment variable to a GitHub authentication token with read access to private repositories (e.g., export GITHUB_TOKEN=<token>). You can generate a GitHub authentication token here.
  3. Set the gh_auth_token global variable in the prior package with:
import prior
prior.gh_auth_token = "<token>"

Citation

To cite the PRIOR package, please use:

@software{prior,
  author={Matt Deitke and Aniruddha Kembhavi and Luca Weihs},
  doi={10.5281/zenodo.7072830},
  title={{PRIOR: A Python Package for Seamless Data Distribution in AI Workflows}},
  url={https://github.com/allenai/prior},
  year={2022}
}

prior's People

Contributors

dependabot[bot] avatar lucaweihs avatar mattdeitke avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

ram81 garfieldnate

prior's Issues

Not able to load houses using prior

Checklist

  • I have verified that the issue exists against the main branch.
  • I have read the relevant section in the contribution guide on reporting bugs.
  • I have checked the issues list for similar or identical bug reports.
  • I have checked the pull requests list for existing proposed fixes.
  • I have checked the CHANGELOG and the commit log to find out if the bug was already fixed in the main branch.
  • I have included in the "Description" section below a traceback from any exceptions related to this bug.
  • I have included in the "Related issues or possible duplicates" section beloew all related issues and possible duplicate issues (If there are none, check this box anyway).
  • I have included in the "Environment" section below the name of the operating system and Python version that I was using when I discovered this bug.
  • I have included in the "Environment" section below the output of pip freeze.
  • I have included in the "Steps to reproduce" section below a minimally reproducible example.

Description

I am trying to load ProctHor's datasets using prior but can't be able to.

Python traceback: 

Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import prior
  File "/home/bnjoroge/.local/lib/python3.8/site-packages/prior/__init__.py", line 18, in <module>
    from prior.utils.types import  DatasetDict
  File "/home/bnjoroge/.local/lib/python3.8/site-packages/prior/utils/types.py", line 10, in <module>
    from attr import field
ImportError: cannot import name 'field' from 'attr' (/usr/lib/python3/dist-packages/attr/__init__.py)

The "field" value is not importable from "attr" and when I traced back to the source it wasn't defined as well. Am I missing something?

I also wasn't able to import the define dcorator.

Related issues or possible duplicates

  • None

Environment

OS:
WSL Ubuntu

Python version:
Python 3.8.1

Output of pip freeze: 

opencv-python==4.6.0.66
packaging==21.3
pandas==1.4.2
parameterized==0.8.1
parso==0.8.3
pathspec==0.9.0
pep517==0.12.0
pexpect==4.6.0
pickleshare==0.7.5
Pillow==9.2.0
pkginfo==1.8.2
platformdirs==2.5.2
pluggy==1.0.0
portalocker==2.5.1
prior==1.0.0
proglog==0.1.10
progressbar2==4.0.0
prompt-toolkit==3.0.29
protobuf==3.19.4
psutil==5.9.1
pure-eval==0.2.2
py==1.11.0
pyarrow==4.0.1
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycairo==1.16.2
pycocotools==2.0.4
pycodestyle==2.8.0
pycosat==0.6.3
pycparser==2.21
pycups==1.9.73
pydantic==1.9.1
pydocstyle==6.1.1
pydot==1.4.2
pyflakes==2.4.0
PyGithub==1.55
Pygments==2.12.0
PyGObject==3.36.0
PyHamcrest==1.9.0
PyHive==0.6.5
PyJWT==2.4.0
pymacaroons==0.13.0
PyNaCl==1.5.0
pyOpenSSL==19.0.0
pyparsing==3.0.9
pyRFC3339==1.1
pyrsistent==0.15.5
pyserial==3.4
PySocks==1.7.1
pytest==7.1.2
pytest-cov==3.0.0
pytest-mock==3.7.0
python-apt==2.0.0+ubuntu0.20.4.5
python-dateutil==2.8.2
python-debian===0.1.36ubuntu1
python-utils==3.3.3
python-xlib==0.31
pytz==2022.1
PyYAML==5.3.1
pyzmq==23.2.0
readme-renderer==35.0
regex==2022.7.25
requests==2.22.0
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
requests-unixsocket==0.2.0
retrying==1.3.3
rfc3986==2.0.0
rich==12.4.4
rsa==4.8
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
s3transfer==0.6.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
service-identity==18.1.0
setuptools-scm==6.4.2
simplejson==3.16.0
six==1.14.0
snowballstemmer==2.2.0
sos==4.1
soupsieve==2.3.2.post1
Sphinx==5.0.0
sphinx-rtd-theme==1.0.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
ssh-import-id==5.10
stack-data==0.2.0
systemd-python==234
tabulate==0.8.10
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
termcolor==1.1.0
testfixtures==6.18.5
timm==0.6.7
toml==0.10.2
tomli==2.0.1
torch==1.10.1+cu111
torch-tb-profiler==0.4.0
torchaudio==0.12.1
torchvision==0.11.2+cu111
tornado==6.2
tqdm==4.64.0
traitlets==5.2.2.post1
twine==4.0.0
Twisted==18.9.0
typing-extensions==4.2.0
ubuntu-advantage-tools==27.2
ufw==0.36
unattended-upgrades==0.1
urllib3==1.26.9
virtualenv==20.14.1
wadllib==1.3.3
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==2.2.1
wrapt==1.14.1
xkit==0.0.0
yacs==0.1.8
zipp==3.8.1
zope.interface==4.7.1

Steps to reproduce

pip install ai2thor, and the dependencies.
Then try to load houses using the load_dataset() method.

Example source: 

pip install --extra-index-url https://ai2thor-pypi.allenai.org ai2thor==0+391b3fae4d4cc026f1522e5acf60953560235971 &> /dev/null
!pip install ai2thor-colab prior --upgrade &> /dev/null

pip install --extra-index-url https://ai2thor-pypi.allenai.org ai2thor==0+391b3fae4d4cc026f1522e5acf60953560235971 &> /dev/null
!pip install ai2thor-colab prior --upgrade &> /dev/null

error in prior

Hi @mattdeitke ,
I use the demo code below to get procthor-10k, but it returns error below.

code:

import prior
prior.load_dataset("procthor-10k")

error:

Loading train:   0%|          | 0/10000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/casia/experiment/AI2THOR/down.py", line 2, in <module>
    prior.load_dataset("procthor-10k")
  File "/home/casia/anaconda3/envs/AI2THOR/lib/python3.9/site-packages/prior/__init__.py", line 247, in load_dataset
    out_dataset: DatasetDict = out["load_dataset"](**params)
  File "<string>", line 44, in load_dataset
  File "<string>", line 44, in <listcomp>
  File "/home/casia/anaconda3/envs/AI2THOR/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/casia/anaconda3/envs/AI2THOR/lib/python3.9/gzip.py", line 398, in readline
    return self._buffer.readline(size)
  File "/home/casia/anaconda3/envs/AI2THOR/lib/python3.9/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/home/casia/anaconda3/envs/AI2THOR/lib/python3.9/gzip.py", line 487, in read
    if not self._read_gzip_header():
  File "/home/casia/anaconda3/envs/AI2THOR/lib/python3.9/gzip.py", line 435, in _read_gzip_header
    raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b've')

No such file or directory: '/media/datc/.git-credentials'

Hi @mattdeitke :
When I use prior to load procthor10k, it returns an error message:

  File "/media/datc/experiment/allenact/allenact_plugins/procthor_plugin/procthor_environment.py", line 76, in __init__
    self.scene_data = prior.load_dataset("procthor-10k")
  File "/media/datc/anaconda3/envs/ai2thor/lib/python3.8/site-packages/prior/__init__.py", line 272, in load_dataset
    sha, token = _clone_repo(
  File "/media/datc/anaconda3/envs/ai2thor/lib/python3.8/site-packages/prior/__init__.py", line 212, in _clone_repo
    with open(f"{os.environ['HOME']}/.git-credentials", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/media/datc/.git-credentials'

Prior package is not found via pip

Hi!
Some days ago I was able to successfully install prior package using pip, but unfortunately since yesterday it's not available any more.

pip install prior
ERROR: Could not find a version that satisfies the requirement prior (from versions: none)
ERROR: No matching distribution found for prior

Can you provide any suggestions or ideas? Thanx!

FileNotFoundError: [Errno 2] No such file or directory: 'wget'

Hi,

When I try to use prior to load procthor-dataset, it throws me the following error:

File "/usr/local/lib/python3.8/dist-packages/prior/__init__.py", line 273, in load_dataset git_lfs_cmd = _get_git_lfs_cmd() File "/usr/local/lib/python3.8/dist-packages/prior/__init__.py", line 92, in _get_git_lfs_cmd subprocess.run( File "/usr/lib/python3.8/subprocess.py", line 493, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'wget'

FileNotFoundError: [Errno 2] No such file or directory: 'wget'

Checklist

  • I have verified that the issue exists against the main branch.
  • I have read the relevant section in the contribution guide on reporting bugs.
  • I have checked the issues list for similar or identical bug reports.
  • I have checked the pull requests list for existing proposed fixes.
  • I have checked the CHANGELOG and the commit log to find out if the bug was already fixed in the main branch.
  • I have included in the "Description" section below a traceback from any exceptions related to this bug.
  • I have included in the "Related issues or possible duplicates" section beloew all related issues and possible duplicate issues (If there are none, check this box anyway).
  • I have included in the "Environment" section below the name of the operating system and Python version that I was using when I discovered this bug.
  • I have included in the "Environment" section below the output of pip freeze.
  • I have included in the "Steps to reproduce" section below a minimally reproducible example.

Description

Python traceback:

Related issues or possible duplicates

  • None

Environment

OS:

Python version:

Output of pip freeze:

Steps to reproduce

Example source:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.