allenai / savn Goto Github PK

Hello, I cannot find the process of updating the parameters in self-supervised loss. On line 125 of the ''savn_train.py'' file, the gradients in self-supervised loss are None, so could you please tell me how to update the parameters? Thank you!

I want to know what the file 'visible_object_map.json' records. In that file, every visible object has a series of states, what does this states mean? Does it means all the states that agent can see the object?

Excuse me, I want to know the distance of visible objects in 'visible_object_map'. What's the threshold distance of visible objects？

I try to use ppo to implement,but the result is worse,maybe something wrong about my code,is there any possiblity to get a ppo/dppo baseline?

Hello, when I run your code, there is an error. Could you give me some advice?

Traceback (most recent call last):
File "main.py", line 16, in
from utils.class_finder import model_class, agent_class, optimizer_class
File "/home/aming/savn/utils/class_finder.py", line 2, in
import agents
File "/home/aming/savn/agents/init.py", line 2, in
from .random_agent import RandomNavigationAgent
File "/home/aming/savn/agents/random_agent.py", line 6, in
from episodes.basic_episode import BasicEpisode
File "/home/aming/savn/episodes/init.py", line 1, in
from .basic_episode import BasicEpisode
File "/home/aming/savn/episodes/basic_episode.py", line 8, in
from datasets.environment import Environment
File "/home/aming/savn/datasets/environment.py", line 7, in
from .offline_controller_with_small_rotation import OfflineControllerWithSmallRotation
File "/home/aming/savn/datasets/offline_controller_with_small_rotation.py", line 653
**self.grid[0], rotation=random.choice(self.rotations)

I would like to ask what is the length, width and height data of the physical attributes of the intelligent robot in the scene in a specific task

Hello, when I call your search_and_save function, what is the cause of the following error? After the error occurs, the process becomes a zombie process, and the program cannot continue.

Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/VT_Depth/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/ubuntu/miniconda3/envs/VT_Depth/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "gen_ground_truth.py", line 39, in search_and_save
c.start()
File "/home/ubuntu/miniconda3/envs/VT_Depth/lib/python3.6/site-packages/ai2thor/controller.py", line 804, in start
self.download_binary()
File "/home/ubuntu/miniconda3/envs/VT_Depth/lib/python3.6/site-packages/ai2thor/controller.py", line 742, in download_binary
os.rename(extract_dir, os.path.join(self.releases_dir(), self.build_name()))
OSError: [Errno 39] Directory not empty: '/home/ubuntu/.ai2thor/tmp/thor-201903131714-Linux64' -> '/home/ubuntu/.ai2thor/releases/thor-201903131714-Linux64'

Hi,

I just found the script in the #3, but it seems that would be asserted at line 384 in offline_controller_with_small_rotation.py.

Meanwhile, I could not understand the visible_object_map.json, some node in that file seems to be further than 1 meter from the target.

Thanks

Hi, have an implementation related question:

In the embedding() method for the basemodel, there is an option for providing params. If params are provided, then the code uses functional Pytorch methods to call the forward passes of the layers: https://github.com/allenai/savn/blob/master/models/basemodel.py#L73

Was there any particular reason to use functional, and not simply load the params into the model state-dict at this phase? I am thinking of replacing the functional calls with a load_state_dict(), because custom layers may not always have a functional PyTorch implementation.... and wanted to make sure this would not break things (I will try it out in anycase...).

Thanks again for your responses (I have been bugging you over most of this year! :D )

Hello，

Thanks for sharing the code.

I encountered an error when training the SAVN model. Here are the commands I entered and the error message:

(savn) wpx@avs:~/project/savn$ python main.py --title savn_train --model SAVN --gpu-ids 0 1 --workers 12
initialized optimizer.
attempting to share memory.
sharing memory.
SAVN(
  (conv1): Conv2d(512, 64, kernel_size=(1, 1), stride=(1, 1))
  (maxp1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (embed_glove): Linear(in_features=300, out_features=64, bias=True)
  (embed_action): Linear(in_features=6, out_features=10, bias=True)
  (pointwise): Conv2d(138, 64, kernel_size=(1, 1), stride=(1, 1))
  (lstm): LSTMCell(3136, 512)
  (critic_linear): Linear(in_features=512, out_features=1, bias=True)
  (actor_linear): Linear(in_features=512, out_features=6, bias=True)
  (action_predict_linear): Linear(in_features=6272, out_features=6, bias=True)
  (dropout): Dropout(p=0.25)
  (ll_key): Linear(in_features=518, out_features=518, bias=True)
  (ll_linear): Linear(in_features=518, out_features=518, bias=True)
  (ll_tc): TemporalConvNet(
	(ll_temporal_block0): TemporalBlock(
	  (ll_conv1): Conv1d(6, 10, kernel_size=(2,), stride=(1,), padding=(1,))
	  (chomp1): Chomp1d()
	  (ll_conv2): Conv1d(10, 10, kernel_size=(2,), stride=(1,), padding=(1,))
	  (chomp2): Chomp1d()
	  (sigmoid): Sigmoid()
	)
	(ll_temporal_block1): TemporalBlock(
	  (ll_conv1): Conv1d(10, 1, kernel_size=(2,), stride=(1,), padding=(2,), dilation=(2,))
	  (chomp1): Chomp1d()
	  (ll_conv2): Conv1d(1, 1, kernel_size=(2,), stride=(1,), padding=(2,), dilation=(2,))
	  (chomp2): Chomp1d()
	  (sigmoid): Sigmoid()
	)
  )
)
Train agents created.
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
	self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
	self._target(*self._args, **self._kwargs)
  File "/home/wpx/project/savn/runners/savn_train.py", line 140, in savn_train
	transfer_gradient_to_shared(meta_gradient, shared_model, gpu_id)
  File "/home/wpx/project/savn/runners/train_util.py", line 110, in transfer_gradient_to_shared
	param._grad = gradient[i].cpu()
RuntimeError: assigned grad has data of a different size

I have checked my gpu and cuda and they are all running normally.

Any suggestions？Thanks.

Hi,

can you please share the file format and structure for storing image features?

It seems your code reads in pre-computed features from an HDF5 dump for each frame that the agent can see in this line in your codebase:

If we want to use other types of features (e.g. a different feature extractor than what you use to represent the frame image), it would be super helpful to have more details on how the feature dump is constructed, the ordering of the frame images, etc.

thank yoU!

Hi,

Thanks for sharing the code. I strictly follow the instruction of this repo, but cannot get the performance reported. I can only get the following performance for SAVN.

{
"GreaterThan/1/spl": 0.13895125447326895,
"GreaterThan/1/success": 0.384,
"GreaterThan/5/spl": 0.11553156460140891,
"GreaterThan/5/success": 0.25898203592814373,
"done_count": 0.974,
"ep_length": 27.018,
"spl": 0.13895125447326895,
"success": 0.384,
"total_reward": 1.6536600000000015,
"total_time": 0.5675430691242218
}

Any suggestion?
Thanks.

Xiaojun

I want to get the image so I downloaded the thor_offlline_data_with_images and followed the docs in h5py's website but there was no output. why? Did i made some mistake?

import h5py
f = h5py.File('images.hdf5', 'r')
list(f.keys())

Hi, I want to know how to generate the graph.json, we want to change the action space and generate new graph.json file.

Hi, it seems that the agent is trained in an offline environment with an offline controller. Are there any instructions to run the trained model in an actual running Ai2Thor simulator?

Hello, I try to train the model in online ai2thor simulator based on your algorithm, but it seems that it is very difficult to get good performance. I use resnet18 to get the feature of every observation the agent can get in ai2thor environment. I have trained the model using online ai2thor simulator, but the overall success rate of navigation is only about 9% in used scenes, when evaluating in unused scene, the performance is worse.

So I want to know the resnet18 structure you used to extracted the feature, maybe it's the problem. I use the pretrained resnet18 to extract the feature of observation, I add a nn.AdaptiveAvgPool2d layer after the conv5_x to get 512 x 7 x 7 feature, and then do action prediction.

Hi,

Is it possible to share any of the visualization scripts? The paper has some really nice figures of the agent navigating around a room, the field of view, etc. Of course, it is possible to plot all this from scratch, but I wondered if you have any visualization scripts handy that can be used with this codebase.

thanks again your helpful responses earlier and for making this codebase publicly available!

Hello, if I want to extract the feature of observation using resnet, how can we extract the features and save to a hdf5 file? Or can you share the resnet structure of the resnet18 you used to extract features?

Hi,

Could you please tell me how to compute the adjacency matrix for gcn baseline. Is the matrix extracted from Visual Genome or from the whole rooms?

Actually, I want to get the object relationships (e.g., the sink is next to the toilet) in a specific scene. However, I only find API to get the objects list in the scene with their centric coordinates. Because of the different scales of objects and coordinates of different rooms, I can not compute the connection relationship of the objects. Do you have any idea?

Thank you very much!

I'm interested in the hand crafted interaction objective, but I don't see any code related. This objective is more intuitive, can you share it?

I tried to run model on the image files, and I add the the command line argument --images_file_name images.hdf5.

But the result of it is RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 512, 1, 1], but got input of size [300, 300, 3] instead

Why it happend? Is there something I haven't done yet? I would be very grateful if I could get a reply

Hello, how did the picture features of those 120 scenes be done? Is there an original picture of 120 scenes?

I am using ai2thor 5.0.0 and torch 1.13.0+cu117 version in a new environment, but I am facing the following error.
AttributeError: module 'torch.nn' has no attribute '_VF'
Please advise on fixing the error.

I just set up this repository following the README docs. The evaluation scripts for Scene Priors and Non-Adaptive A3C worked, but not for SAVN. Because pytorch 0.4.1 is out of date, I tried with 1.8.1. I needed to change nn._VF.lstm_cell to torch._VF.lstm_cell. Then it worked. Just posting this here in case anyone runs into this issue.

The following code also works (replace the torch._VF call:

lstm_cell = nn.quantizable.LSTMCell.from_params(params["lstm.weight_ih"],
                                                params["lstm.weight_hh"],
                                                bi=params["lstm.bias_ih"],
                                                bh=params["lstm.bias_hh"])
hx, cx = lstm_cell(embedding, prev_hidden)

Hi,
How long will it take to train a model as good as your pretrained ones or achieve the result reported in readme?

           thank you~

I want to train + test the model on 60 categories (rather than 19 used in savn). Could you please tell what algorithm you used to generate test-set data in test_val_split? Or did you generate it randomly?

Hi,

In the thor offline data folder (data/thor_offline_data or data/mixed_offline_data), 60 of the floorplans do not have resnet50_fc.hdf5 files, e.g. "FloorPlan304". However, all 120 have resnet18_featuremap.hdf5 files.

Can you please share the resnet50_fc.hdf5 features for all scenes?

Hi,

I'm having trouble downloading the thor_offline_data_with_images. The link doesn't download anything. Can you please provide another way of downloading the data?

The other data.tar.gz file was downloaded from the same s3 bucket without an issue. So I'm not sure what the problem might be for thor_offline_data_with_images.

Thanks!

Excuse me. I want to know how do you get the glove embedding of the target objects? Do you use some model to extract the glove embedding?