facebookresearch / r3m Goto Github PK

View Code? Open in Web Editor NEW

266.0 14.0 44.0 41 KB

Pre-training Reusable Representations for Robotic Manipulation Using Diverse Human Video Data

Home Page: https://sites.google.com/view/robot-r3m/

License: MIT License

Python 100.00%

r3m's People

Stargazers

Watchers

r3m's Issues

Questions about clip preprocessing

The ' narrations.json' file published by EGO4D does not provide the start time and end time of each clip, only the timestamp_frame:

First, for each narration_pass, I sort all clips by 'timestamp_frame'. Then,for the current clip, the 'timestamp_frame 'is the start frame, and the 'timestamp_frame' of the next clip is the end frame. However, there are clips longer than 5 minutes in the preprocessed data. This is different from the explanation "each clip was about 200 frames, so about 10 seconds" mentioned in #13.

Is this clip segmentation appropriate? When filtering too long and too short data, what is the maximum and minimum number of frames you set?

No "camera" with name left_cap2 exists. Available "camera" names = ('topview', 'corner', 'corner2', 'corner3', 'behindGripper', 'gripperPOV').

Hi,
When I try to run the ""hammer-v2-goal-observable", "left cap2" on metaworld, it reporters the above error.
But the code runs well on "kitchen_sdoor_open-v3" with another environment.
Is there anything setting I miss or should I change some configs?

Evaluation results are always 0% success rates.

Hi, thanks for your great work. I have configured all the environment depended by Franka Kitchen and Adroit about a month ago, and training results, and the training results look similar to the paper. However, recently I reconfigured the environment and ran the training command. The losses decreased normally, but the success rates are always 0.0%, whether on the kitchen_sdoor_open-v3 task or the pen-v0 task. I have tried my best to figure out this problem, but finally failed. Is this related to the recent updates to the mj_evns repository? Hope for help.

How to apply R3M to downstream robotic task?

Hi,

in the paper, it was mentioned that when we apply R3M to downstream robot learning tasks, only a few demonstrations are needed. My question is that do we also need to add language in these demos? Or only video demos are needed?

Ego4d preprocessing steps

First congratulations on your paper and thanks for sharing the codebase! I am recently studying some of the implementation details and trying to reproduce the R3M training on Ego4D. I am wondering about if there are some preprocessing steps on the Ego4D dataset that have not been released. For instance, I think the paper suggests to split the video into individual frames (are the videos used a subset of the full_scale Ego4D) and the codebase suggests to use 'len' and "txt" key in the manifest file (these fields seem to be missing). It would be great if you can hint on how to preprocess the dataset or point to related code that I might have missed. Thanks!

Missing model.pt File

Recently I wanted to see the representation of R3M model. But it seems that the checkpoints file is missing.

FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.r3m/r3m_50/model.pt'

How am I supposed to get to this file?

How to recover text grounding from visual encoder

Hey, thx so much for sharing this repo!
since r3m is trained via contrastive learning, it should have learned to align the visual representation to text embeddings. So based on this, I do wonder if is there any efficient approach that when using r3m, for a given visual representation, we can further decode its textual grounding.
I think one approach is to use a pre-trained captioning model to generate captions, then further infer the description, what do you think of it?

Pre-allocated constraint buffer is full. Increase njmax above 1000.

Hi, thanks for your excellent work. When I run command to train R3M on kitchen_sdoor_open-v3 environment, it works fine at the beginning, but at a certain iteration it throws an error:

>>> python hydra_launcher.py hydra/launcher=local hydra/output=local env=kitchen_sdoor_open-v3 camera=left_cap2 pixel_based=true embedding=resnet50 num_demos=5 env_kwargs.load_path=r3m bc_kwargs.finetune=false bc_kwargs.save_logs=true proprio=9 seed=125 job_name=kitchen_sdoor_open-v3_left-cap2

...
Step 3955
Step 3962
Step 3969
Step 3976
Step 3983
Step 3990
Step 3997
Evaluating
Error executing job with overrides: ['env=kitchen_sdoor_open-v3', 'camera=left_cap2', 'pixel_based=true', 'embedding=resnet50', 'num_demos=5', 'env_kwargs.load_path=r3m', 'bc_kwargs.finetune=false', 'bc_kwargs.save_logs=true', 'proprio=9', 'seed=125', 'job_name=kitchen_sdoor_open-v3_left-cap2']
An error occurred during Hydra's exception formatting:
AssertionError()
Traceback (most recent call last):
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/_internal/utils.py", line 252, in run_and_report
    assert mdl is not None
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hydra_launcher.py", line 42, in <module>
    configure_jobs()
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main
    config_name=config_name,
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra
    lambda: hydra.run(
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/_internal/utils.py", line 294, in run_and_report
    raise ex
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in <lambda>
    overrides=args.overrides,
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/tiger/.local/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "hydra_launcher.py", line 38, in configure_jobs
    bc_train_loop(job_data)
  File "/opt/tiger/Robotics/r3m/evaluation/r3meval/core/train_loop.py", line 151, in bc_train_loop
    env_kwargs=env_kwargs)
  File "/opt/tiger/Robotics/r3m/evaluation/r3meval/utils/sampling.py", line 172, in sample_paths
    return do_rollout(**input_dict)
  File "/opt/tiger/Robotics/r3m/evaluation/r3meval/utils/sampling.py", line 95, in do_rollout
    o = env.reset()
  File "/opt/tiger/Robotics/r3m/evaluation/r3meval/utils/gym_env.py", line 91, in reset
    return self.env.reset()
  File "/home/tiger/.local/lib/python3.7/site-packages/gym/core.py", line 311, in reset
    return self.observation(self.env.reset(**kwargs))
  File "/home/tiger/.local/lib/python3.7/site-packages/gym/core.py", line 311, in reset
    return self.observation(self.env.reset(**kwargs))
  File "/home/tiger/.local/lib/python3.7/site-packages/gym/wrappers/time_limit.py", line 26, in reset
    return self.env.reset(**kwargs)
  File "/home/tiger/.local/lib/python3.7/site-packages/gym/wrappers/order_enforcing.py", line 18, in reset
    return self.env.reset(**kwargs)
  File "/opt/tiger/Robotics/lib/mj_envs/mj_envs/envs/relay_kitchen/kitchen_multitask_v2.py", line 335, in reset
    self.sim.step()
  File "mjsim.pyx", line 126, in mujoco_py.cymj.MjSim.step
  File "cymj.pyx", line 156, in mujoco_py.cymj.wrap_mujoco_warning.__exit__
  File "cymj.pyx", line 77, in mujoco_py.cymj.c_warning_callback
  File "/home/tiger/.local/lib/python3.7/site-packages/mujoco_py/builder.py", line 354, in user_warning_raise_exception
    raise MujocoException(warn + 'Increase njmax in mujoco XML')
mujoco_py.builder.MujocoException: Pre-allocated constraint buffer is full. Increase njmax above 1000. Time = 4.0040.Increase njmax in mujoco XML

I guess it may be caused by the buffer constrain of Mujoco. I have found parameter njmax related files under the fold ~/.mujoco/mujoco200/model, but I have no idea which file should be modified. Do you have any advice?

Encounter some issue with hydra multirun

Hi, I succeeded in verifying my installation for evaluation. However when I tried to reproduce your complete visuomotor control results, I encountered with some issue with hydra multirun. It seems to be something with the hydra configs. Could you please tell me how to modify them accordingly? thx a lot~

RS:> Registering Biomechanics Envs
RS:> Registering Hand Envs
RS:> Registering Kitchen Envs
Traceback (most recent call last):
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra/_internal/utils.py", line 386, in <lambda>
    lambda: hydra.multirun(
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 140, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 157, in sweep
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 146, in launch
    return [j.results()[0] for j in jobs]
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 146, in <listcomp>
    return [j.results()[0] for j in jobs]
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/submitit/core/core.py", line 294, in results
    raise job_exception  # pylint: disable=raising-bad-type
submitit.core.utils.FailedJobError: Job (task=0) failed during processing with trace:
----------------------
Traceback (most recent call last):
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/pathlib.py", line 1323, in mkdir
    self._accessor.mkdir(self, mode)
OSError: [Errno 36] File name too long: 'outputs/BC_pretrained_rep/2023-11-29_10-46-23/0_bc_kwargs.finetune=False,camera=[left_cap2,right_cap2],embedding=resnet50,env=[kitchen_knob1_on-v3,kitchen_light_on-v3,kitchen_sdoor_open-v3,kitchen_ldoor_open-v3,kitchen_micro_open-v3],env_kwargs.load_path=r3m,job_name=try_r3m_franka_kitchen,num_demos=25,pixel_based=True,proprio=9'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/submitit/core/submission.py", line 55, in process_job
    result = delayed.result()
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/submitit/core/utils.py", line 133, in result
    self._result = self.function(*self.args, **self.kwargs)
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 71, in __call__
    return run_job(
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/site-packages/hydra/core/utils.py", line 145, in run_job
    Path(str(working_dir)).mkdir(parents=True, exist_ok=True)
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/pathlib.py", line 1332, in mkdir
    if not exist_ok or not self.is_dir():
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/pathlib.py", line 1439, in is_dir
    return S_ISDIR(self.stat().st_mode)
  File "/home/wangbangjun/anaconda3/envs/r3m_base/lib/python3.9/pathlib.py", line 1232, in stat
    return self._accessor.stat(self)
OSError: [Errno 36] File name too long: 'outputs/BC_pretrained_rep/2023-11-29_10-46-23/0_bc_kwargs.finetune=False,camera=[left_cap2,right_cap2],embedding=resnet50,env=[kitchen_knob1_on-v3,kitchen_light_on-v3,kitchen_sdoor_open-v3,kitchen_ldoor_open-v3,kitchen_micro_open-v3],env_kwargs.load_path=r3m,job_name=try_r3m_franka_kitchen,num_demos=25,pixel_based=True,proprio=9'

hydra error in r3meval

Hi,
Thanks for bringing such a great codebase.
I just came across an error in R3Meval

haoyux@robot01:~/r3m-bc/evaluation/r3meval/core$ python hydra_launcher.py --multirun hydra/launcher=local hydra/output=local env=["drawer-open-v2-goal-observable"] camera=["top_cap2"] pixel_based=true embedding=resnet50 num_demos=25 env_kwargs.load_path=r3m bc_kwargs.finetune=false proprio=4 job_name=try_r3m
/home/haoyux/anaconda3/envs/robotube/lib/python3.8/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
Error executing job with overrides: ['env=[drawer-open-v2-goal-observable]', 'camera=[top_cap2]', 'pixel_based=True', 'embedding=resnet50', 'num_demos=25', 'env_kwargs.load_path=r3m', 'bc_kwargs.finetune=False', 'proprio=4', 'job_name=try_r3m']
Traceback (most recent call last):
File "/home/haoyux/anaconda3/envs/robotube/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/haoyux/anaconda3/envs/robotube/lib/python3.8/site-packages/hydra/_internal/utils.py", line 386, in <lambda>
lambda: hydra.multirun(
File "/home/haoyux/anaconda3/envs/robotube/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 140, in multirun
ret = sweeper.sweep(arguments=task_overrides)
File "/home/haoyux/anaconda3/envs/robotube/lib/python3.8/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 161, in sweep
_ = r.return_value
File "/home/haoyux/anaconda3/envs/robotube/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
TypeError: can only concatenate str (not "ListConfig") to str

Imitation Evaluation Code

Hi, thank you for releasing such a wonderful work.
When the imitation learning evaluation code be released?

Can not run evaluation on V100 GPU

Thank you for your great work,
However, I have some problems when running your evaluation code.
My GPU enviroment is Tesla V100-32 GPU, and it always reports such rendering error:

Found 0 GPUs for rendering. Using device 0.
File "mjsim.pyx", line 156, in mujoco_py.cymj.MjSim.render
  File "mjsim.pyx", line 158, in mujoco_py.cymj.MjSim.render
  File "mjrendercontext.pyx", line 46, in mujoco_py.cymj.MjRenderContext.__init__
  File "mjrendercontext.pyx", line 114, in mujoco_py.cymj.MjRenderContext._setup_opengl_context
  File "opengl_context.pyx", line 130, in mujoco_py.cymj.OffscreenOpenGLContext.__init__
RuntimeError: Failed to initialize OpenGL

The bug happens here in L184-L191 of obs_wrappers.py

def get_image(self):
        if self.camera_name == "default":
            img = self.sim.render(width=self.width, height=self.height, depth=self.depth,
                            device_id=self.device_id)
        else:
            img = self.sim.render(width=self.width, height=self.height, depth=self.depth,
                              camera_name=self.camera_name, device_id=self.device_id)
        img = img[::-1,:,:]
        return img

So I wonder what GPU enviroment do you use in your evaluation experiments? Is it the problem of GPU type or the problem of my installation mistake?

Ego4D manifest/clip length

Hi, thanks for releasing your code! Would you be able to release the manifest file you used? In particular, the narrations file in the Ego4D data does not provide lengths for each of the fine-grained narrations. If you have any alternative recommendation for how to get the lengths, I'd appreciate that. Thanks!

Imitation train time

Hi, I execute the following command on a single A100, but the GPU utilization is only 0%~9%. And it takes 3 days to complete the 20000 steps training.
python hydra_launcher.py hydra/launcher=local hydra/output=local env="kitchen_sdoor_open-v3" camera="left_cap2" pixel_based=true embedding=resnet50 num_demos=5 env_kwargs.load_path=r3m bc_kwargs.finetune=false proprio=9 job_name=r3m_repro seed=125
Is such GPU utilization normal? How long do you spend training downstream tasks?

Thanks

Release Ego4D Preprocessing code?

Thnaks for your excellent work! Could you please release Ego4D preprocessing code to facilitate reproducing and developing your work?

how much gpu resource for pretraining

Thank you for this wonderful work.Can you tell me the gpu resource and time you spend

Training process goes well, but evaluation results are either 0% or 100 % success rates.

Hi, thanks for your awesome work. I follow the steps in README and successfully train the policy network. The loss decreases to a very low level. However, as shown in figure, the evaluation results show that max success rates are either 0% or 100% on benchmarks like kitchen_light_on-v3, kitchen_ldoor_open-v3 and kitchen_sdoor_open-v3. Is this reasonable? Or do you have any suggestions?

Reproducing Moco345/ImNet results

Hi,

I am trying to reproduce your results using other backbones than yours,

However when using Moco345 (weights taken from https://github.com/sparisi/pvr_habitat/releases/tag/models; moco_crop, moco_crop_l4, moco_crop_l3) I obtain an average success rate (using 25 demos) of 28% while when I use pretrained pytorch model I get 34%.

Based on your article, I expected to get the opposite results. Could you clarify this point? (I used exactly the same code with a new Conda environment and consistent results using your model).

Thanks

Question about image sampling for r3m training.

Hi, thanks for your impressive work. Recently I'm trying to retrain r3m model and confused about the sampling of images for contrastive learning. If I understand correctly, according to the code in data_loaders.py, taking vidlen=30 and alpha=0.2 as an example, then start_in is in [1, 8], end_ind in [23, 30], s1_ind in [2, 30], s0_ind in [1, s1_ind] and s2_ind in [s1_ind, 31]. Assuming that start_in=7, end_ind=25, s1_ind=6, s0_ind=2 and s2_ind=20, then positive image-language pair of the second example in trainer.py is {img7, img6, text}, and one of the negative pair is {img7, img2, text}. I'm confused about this sampling result. In my opinion, transitioning from img7 to img2 completes the language text better than one from img7 to img6, and the former should be positive example rather than negative example. Hope for help.

Can the robot achieve a certain task from man without language?

Hi, I am interested in this work. In the issues #5, you mentioned that the pre-trained R3M model simply acts as an encoder mapping images to embeddings. My question is how to use the whole framework in the downstream robotic task after behavior cloning? Just given an image, then the robot can do the imitated work? How about in a more sophisticated environment? How can the robot achieve a certain task from man without language?

Relation of frame indices

From the lines here

r3m/r3m/utils/data_loaders.py

Line 77 in b2334e7

s1_ind = np.random.randint(2, vidlen)

, it seems possible for frame im0 to be after frames imts0/imts1/imts2 (or some subset thereof). Is this not a problem for the loss terms where we want to train G(e_0, e_t, l) to be higher than G(e_0, e_<t, l) and G(e*0, e*<t, l)?

TypeError: can only concatenate str (not "ListConfig") to str

Hi, thanks for your awesome work. The commands for kitchen environment run well and training results are also reasonable. However, the command of Adroit pen task fails to run in my case.

>>> python hydra_launcher.py --multirun hydra/launcher=local hydra/output=local env=pen-v0 camera=["view_1","top","view_4"] pixel_based=true embedding=resnet50 num_demos=25 env_kwargs.load_path=r3m bc_kwargs.finetune=false proprio=24 job_name=try_r3m
/root/miniconda3/envs/r3m/lib/python3.7/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
RS:> Registering Arms Envs
/root/miniconda3/envs/r3m/lib/python3.7/site-packages/transforms3d/quaternions.py:26: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  _MAX_FLOAT = np.maximum_sctype(np.float)
RS:> Registering Biomechanics Envs
RS:> Registering Hand Envs
RS:> Registering Kitchen Envs
/root/miniconda3/envs/r3m/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py:3: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):
Error executing job with overrides: ['env=pen-v0', 'camera=[view_1,top,view_4]', 'pixel_based=True', 'embedding=resnet50', 'num_demos=25', 'env_kwargs.load_path=r3m', 'bc_kwargs.finetune=False', 'proprio=24', 'job_name=try_r3m']
Traceback (most recent call last):
  File "/root/miniconda3/envs/r3m/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/root/miniconda3/envs/r3m/lib/python3.7/site-packages/hydra/_internal/utils.py", line 389, in <lambda>
    overrides=args.overrides,
  File "/root/miniconda3/envs/r3m/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 140, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "/root/miniconda3/envs/r3m/lib/python3.7/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 161, in sweep
    _ = r.return_value
  File "/root/miniconda3/envs/r3m/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
TypeError: can only concatenate str (not "ListConfig") to str

The versions of the relevant packages are as follows:

hydra-core==1.1.1
hydra-submitit-launcher==1.1.5
omegaconf==2.1.1

They are the same as in your r3m_base.yaml. I also have tried other versions, but the bug is always there. Do you have any suggestions?

Encoding image batches

Hey, thanks a lot for open-sourcing this! I've seen that you guys have an example for a single image here but I'm wondering what kind of image dimensions can be passed to the pre-trained models and if they can work with image batches (e.g. (num_envs, 3, H, W)). Thanks!

Fail to load franka kitchen related environments

Hi, I have successfully configured all environments according to your instructions before and all commands run well. However, I reconfigured all environments in another machine recently and following error always encounters:

>>> python hydra_launcher.py hydra/launcher=local hydra/output=local env=kitchen_sdoor_open-v3 camera=left_cap2 pixel_based=true embedding=resnet50 num_demos=5 env_kwargs.load_path=r3m bc_kwargs.finetune=false bc_kwargs.save_logs=true proprio=9 seed=125 job_name=kitchen_sdoor_open-v3_left-cap2
/root/miniconda3/envs/r3m/lib/python3.7/site-packages/gym/spaces/box.py:112: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
RS:> Registering Arms Envs
/root/miniconda3/envs/r3m/lib/python3.7/site-packages/transforms3d/quaternions.py:26: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  _MAX_FLOAT = np.maximum_sctype(np.float)
RS:> Registering Biomechanics Envs
RS:> Registering Hand Envs
RS:> Registering Kitchen Envs
========================================
Job Configuration
========================================
default:
- override hydra/launcher: local
- override hydra/output: local
env: kitchen_sdoor_open-v3
pixel_based: true
embedding: resnet50
camera: left_cap2
device: cuda
seed: 125
steps: 20000
eval_frequency: 1000
eval_num_traj: 50
num_cpu: 1
num_demos: 5
proprio: 9
env_kwargs:
  env_name: ${env}
  device: ${device}
  image_width: 256
  image_height: 256
  camera_name: ${camera}
  embedding_name: ${embedding}
  pixel_based: ${pixel_based}
  render_gpu_id: 0
  load_path: r3m
  proprio: ${proprio}
  lang_cond: false
  gc: false
reparse_kwargs:
  visualize: true
  save_frames: true
bc_kwargs:
  loss_type: MSE
  batch_size: 32
  lr: 0.001
  save_logs: true
  finetune: false
  proprio: ${proprio}
  proprio_only: false
job_name: kitchen_sdoor_open-v3_left-cap2
cwd: /root/Robotics/r3m/evaluation/r3meval/core

5
Demonstration score : 11.05

Traceback (most recent call last):
  File "wrappers.pxi", line 1137, in mujoco_py.cymj.PyMjModel._extract_mj_names
AssertionError
Exception ignored in: 'mujoco_py.cymj.PyMjModel._set'
Traceback (most recent call last):
  File "wrappers.pxi", line 1137, in mujoco_py.cymj.PyMjModel._extract_mj_names
AssertionError
Segmentation fault (core dumped)

However, the commands regarding MetaWorld and Adroit environments run well. I suspect that the Franka Kitchen environment is not registered successfully. Do you have any suggestions?

Pinned old versions of PyTorch in setup.py

Hi,
As I was trying to set up this repo, I realized that some dependencies are pinned to a version that is significantly older than the current versions, for example torch==1.7.1 and torchvision==0.8.2. However, pinning such an old version makes this library incompatible with newer CUDA versions such as 11.3, which makes it hard to use this model on the latest consumer GPUs, like [1].

Is there a specific reason why the dependency is pinned this way? Would you consider relaxing this requirement if we were to test and ensure that the code works without a problem with the later versions of PyTorch?
Thank you for your time!
Mahi

[1]: I am using an nvidia 3080, for example.

Error with using mujoco_py to load xml model

Hi,

Thanks for the nice work!

I'm trying to reproduce the evaluation results. And got the error shown below after getting everything in the env installed (python3.7 + mujoco200 + mujoco_py=2.0.2.13) and data prepared:

But actually I've got this xml file existing with the referred full path. And it seems like the problem with the version of mujoco_py in other cases but updating it causes conflicts with the installed metaworld env requirements.

I'm wondering if you have ever met this problem before and any suggestions?

Best,
Yilin

Access to the language&vision module pre-trained weights

Hi,
Can we have access to the weights of the trained language&vision module (pre-trained weights with "lang_enc" keys), so that I can use the pre-trained R3M with the language head (removing the call to "remove_language_head()" function in the load_r3m() function).
Thanks,

Do we have a vit version r3m pretrained model, besides resnet?

I want to deal with a video sequence using r3m model, but I can only find resnet pretrained model. If there is only resnet model, how can the model deal with a video sequence(what's the input and output)

AttributeError: 'KitchenFrankaRandomDesk' object has no attribute 'reset_model'

run

python hydra_launcher.py hydra/launcher=local hydra/output=local env="kitchen_sdoor_open-v3" camera="left_cap2" pixel_based=true embedding=resnet50 num_demos=5 env_kwargs.load_path=r3m bc_kwargs.finetune=false proprio=9 job_name=r3m_repro seed=125

get output:

Traceback (most recent call last):
  File "/home/pi/workspace/tanwenxuan/Project/r3m/evaluation/r3meval/utils/gym_env.py", line 88, in reset
    return self.env.unwrapped.reset_model(seed=seed)
AttributeError: 'KitchenFrankaRandomDesk' object has no attribute 'reset_model'

About clips data processing

I really appreciate your wonderful work and nice idea！
I'm now faced with some problems when trying to extract ego4d (clip - text) pairs data.
The narration.json only has "timestamp_sec", "timestamp_frame" of a specific clip without "start_time" and "end_time", I wonder how do you decide the interval of the clips?
My method is reranking the clips narration by "timestamp_sec" , and decide the interval of the i-th clip is just [i-th timestamp_sec,i+1-th timestamp_sec]. Is it correct?Does anyone know about it?
Thanks.

facebookresearch / r3m Goto Github PK

r3m's People

Stargazers

Watchers

Forkers

r3m's Issues

Recommend Projects

Recommend Topics

Recommend Org