facebookresearch / mtenv Goto Github PK

View Code? Open in Web Editor NEW

74.0 7.0 5.0 197 KB

MultiTask Environments for Reinforcement Learning.

License: MIT License

Python 99.21% Makefile 0.19% Batchfile 0.24% TeX 0.36%

mtenv's Introduction

MTEnv

MultiTask Environments for Reinforcement Learning.

Introduction
Installation
Usage
Documentation
Contributing to MTEnv
Community
Acknowledgements

Introduction

MTEnv is a library to interface with environments for multi-task reinforcement learning. It has two main components:

A core API/interface that extends the gym interface and adds first-class support for multi-task RL.
A collection of environments that implement the API.

Together, these two components should provide a standard interface for multi-task RL environments and make it easier to reuse components and tools across environments.

You can read more about the difference between MTEnv and single-task environments here.

List of publications & submissions using MTEnv (please create a pull request to add the missing entries):

License

Citing MTEnv

If you use MTEnv in your research, please use the following BibTeX entry:

@Misc{Sodhani2021MTEnv,
  author =       {Shagun Sodhani and Ludovic Denoyer and Pierre-Alexandre Kamienny and Olivier Delalleau},
  title =        {MTEnv - Environment interface for mulit-task reinforcement learning},
  howpublished = {Github},
  year =         {2021},
  url =          {https://github.com/facebookresearch/mtenv}
}

Installation

MTEnv has two components - a core API and environments that implement the API.

The Core API can be installed via pip install mtenv or pip install git+https://github.com/facebookresearch/mtenv.git@main#egg=mtenv

The list of environments, that implement the API, is available here. Any of these environments can be installed via pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[env_name]". For example, the MetaWorld environment can be installed via pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[metaworld]".

All the environments can be installed at once using pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[all]". However, note that some environments may have incompatible dependencies.

MTEnv can also be installed from the source by first cloning the repo (git clone [email protected]:facebookresearch/mtenv.git), cding into the directory cd mtenv, and then using the pip commands as described above. For example, pip install mtenv to install the core API, and pip install "mtenv[env_name]" to install a particular environment.

Usage

MTEnv provides an interface very similar to the standard gym environments. One key difference between multi-task environments (that implement the MTEnv interface) and single-task environments is in terms of observation that they return.

MultiTask Observation

The multi-task environments return a dictionary as the observation. This dictionary has two keys: (i) env_obs, which maps to the observation from the environment (i.e., the observation that a single task environments return), and (ii) task_obs, which maps to the task-specific information from the environment. In the simplest case, task_obs can be an integer denoting the task index. In other cases, task_obs can provide richer information.

from mtenv import make
env = make("MT-MetaWorld-MT10-v0")
obs = env.reset()
print(obs)
# {'env_obs': array([-0.03265039,  0.51487777,  0.2368754 , -0.06968209,  0.6235982 ,
#    0.01492813,  0.        ,  0.        ,  0.        ,  0.03933976,
#    0.89743189,  0.01492813]), 'task_obs': 1}
action = env.action_space.sample()
print(action)
# array([-0.76422   , -0.15384133,  0.74575615, -0.11724994], dtype=float32)
obs, reward, done, info = env.step(action)
print(obs)
# {'env_obs': array([-0.02583682,  0.54065546,  0.22773503, -0.06968209,  0.6235982 ,
#    0.01494118,  0.        ,  0.        ,  0.        ,  0.03933976,
#    0.89743189,  0.01492813]), 'task_obs': 1}

Documentation

https://mtenv.readthedocs.io

Contributing to MTEnv

There are several ways to contribute to MTEnv.

Use MTEnv in your research.
Contribute a new environment. We support many environments via MTEnv and are looking forward to adding more environments. Contributors will be added as authors of the library. You can learn more about the workflow of adding an environment here.
Check out the good-first-issues on GitHub and contribute to fixing those issues.
Check out additional details here.

Community

Ask questions in the chat or github issues:

Chat
Issues

Acknowledgements

Project file pre-commit, mypy config, towncrier config, circleci etc are based on same files from Hydra.

mtenv's People

Contributors

Stargazers

Watchers

Forkers

rohitkuk odelalleau hannahaih chauncygu

mtenv's Issues

Upgrade Dependencies

materials.xml is missing

Copy-pasted from facebookresearch/mtrl#3

Reported by @Leonardo-H

Description

Hi, I meet with some missing file problem when I try to run exps with mtrl.

After installation according to the instructions, when I try to run deepMDP or HiP-BMDP algorithms, there is an error about the following XML file is missing:

/home/myname/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml

I wonder if I miss some packages which should be installed before running experiments?

How to reproduce

create a new python env with anaconda:

conda create -n test python==3.8

run the following commands to install mtrl/mtenv:

git clone https://github.com/facebookresearch/mtrl.git
cd mtrl
pip install -r requirements/dev.txt
pip install mtenv
pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[metaworld]"
pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[hipbmdp]"
pip install dm_control
pip install --upgrade numpy

and use the following to start the experiments (according to https://mtrl.readthedocs.io/en/latest/pages/tutorials/baseline.html):

MUJOCO_GL="osmesa" LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/nvidia-opengl/:$LD_LIBRARY_PATH PYTHONPATH=. python3 -u main.py \
setup=hipbmdp \
env=dmcontrol-finger-spin-distribution-v0 \
agent=hipbmdp \
setup.seed=1 \
replay_buffer.batch_size=256

Error Messages:

torch version = 1.7.1+cu110
/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
[2021-04-08 03:35:38,823][absl][INFO] - MUJOCO_GL=osmesa, attempting to import specified OpenGL backend.
[2021-04-08 03:35:38,830][OpenGL.acceleratesupport][INFO] - No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[2021-04-08 03:35:39,000][absl][INFO] - MuJoCo library version is: 200
Traceback (most recent call last):
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 66, in call
    return _call_callable(type_or_callable, config, *args, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 510, in _call_callable
    return fn(*args, **final_kwargs)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/builder.py", line 46, in build_dmcontrol_vec_env
    env = VecEnv(env_metadata=env_metadata, env_fns=funcs_to_make_envs, context="spawn")
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/vec_env.py", line 22, in __init__
    super().__init__(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/vector/async_vector_env.py", line 77, in __init__
    dummy_env = env_fns[0]()
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/env/builder.py", line 32, in _func
    return mtenv.make(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/registration.py", line 78, in make
    env = mtenv_registry.make(id, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
    env = spec.make(**kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 60, in make
    env = cls(**_kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/env.py", line 78, in build
    mtenv = MultiEnvWrapper(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/shared/wrappers/multienv.py", line 44, in __init__
    self._envs[initial_task_state] = funcs_to_make_envs[initial_task_state]()
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/env.py", line 58, in _func
    return dmc_env.build_dmc_env(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/dmc_env.py", line 96, in build_dmc_env
    env = _build_env(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/dmc_env.py", line 56, in _build_env
    return gym.make(env_id)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 145, in make
    return registry.make(id, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
    env = spec.make(**kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 59, in make
    cls = load(self.entry_point)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/gym/envs/registration.py", line 18, in load
    mod = importlib.import_module(mod_name)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/mtenv/envs/hipbmdp/wrappers/dmc_wrapper.py", line 10, in <module>
    import local_dm_control_suite as local_dmc_suite
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/__init__.py", line 27, in <module>
    from . import (
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/acrobot.py", line 27, in <module>
    from . import common
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/common/__init__.py", line 33, in <module>
    ASSETS = {
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/common/__init__.py", line 34, in <dictcomp>
    filename: resources.GetResource(os.path.join(_SUITE_DIR, filename))
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/dm_control/utils/io.py", line 22, in GetResource
    with open(name, mode=mode) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 63, in call
    return _instantiate_class(type_or_callable, config, *args, **kwargs)
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 500, in _instantiate_class
    return clazz(*args, **final_kwargs)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/dmcontrol.py", line 23, in __init__
    super().__init__(config, experiment_id)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/multitask.py", line 25, in __init__
    super().__init__(config, experiment_id)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/experiment.py", line 33, in __init__
    self.envs, self.env_metadata = self.build_envs()
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/multitask.py", line 43, in build_envs
    envs[mode] = hydra.utils.instantiate(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.env.builder.build_dmcontrol_vec_env' : [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "main.py", line 15, in launch
    return run(config)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/app/run.py", line 35, in run
    experiment_utils.prepare_and_run(config=config)
  File "/home/v-jiahuang/test_mtrl/mtrl/mtrl/experiment/utils.py", line 23, in prepare_and_run
    experiment = hydra.utils.instantiate(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/utils.py", line 70, in call
    raise HydraException(f"Error calling '{cls}' : {e}") from e
hydra.errors.HydraException: Error calling 'mtrl.experiment.dmcontrol.Experiment' : Error calling 'mtrl.env.builder.build_dmcontrol_vec_env' : [Errno 2] No such file or directory: '/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/local_dm_control_suite/./common/materials.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 19, in <module>
    launch()
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/home/v-jiahuang/anaconda3/envs/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
    assert mdl is not None
AssertionError

System information

MTRL Version : commit eea3c99cc116e0fadc41815d0e7823349fcc0bf4
MTRL environment Name : dmcontrol-finger-spin-distribution-v0
python=3.8

Any other information

Btw, when I install mtrl, there is another error/warning:

 FileNotFoundError: [Errno 2] No such file or directory: 'requirements/main.txt'

and indeed I can not find main.txt in requirements, but the installation is not interrupted so I guess maybe it's not an important problem.

metaclass conflict

Hello

Right after installation of the mtenv, when importing into python (for instance running 'from mtenv import make'), the following error appears:

Traceback (most recent call last):
File "", line 1, in
File "/home/victor/Downloads/mtenv-1.0.0/mtenv/init.py", line 4, in
from mtenv.core import MTEnv # noqa: F401
File "/home/victor/Downloads/mtenv-1.0.0/mtenv/core.py", line 21, in
class MTEnv(Env, ABC): # type: ignore[misc]
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

using python 3.9

the error was obtained following the following steps:

pip install git+https://github.com/facebookresearch/mtenv.git@main#egg=mtenv
pip install git+https://github.com/facebookresearch/mtenv.git@main#egg="mtenv[metaworld]"
python3.9

from mtenv import make

Do you have an idea of what is happening?
Thanks for your help?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.