avishkarsaha / translating-images-into-maps Goto Github PK

Hello！Thanks for your good work ! Could you please explain what GPU you used to train models, and how long does it take?

I downloaded the source code and nuscene data(mini version, provided by the Author) as README.txt told, after training 1000 epochs:

1. if set data_size = 0.2, the val_loss.txt will always be around 35
2. if set data_size - 0.7 the val_loss.txt will always be around 20

whatever the value of data_size is, any input image will get an nonsense output. I tried several different images(from nuscene or mine) but the result images look similar. I have no ideal where is the problem.

So have anyone succeeded in training a model to get a good result?

Hi @avishkarsaha ,

I think there is a bug in the above referenced line of the make_grid2d() function that you invoke in the dataloader. For the grid to be consistent with the tensor slicing implemented here, I believe that the order of zz and xx should be reversed. In short, zz, xx = torch.meshgrid(zcoords, xcoords) should be replaced with xx, zz = torch.meshgrid(xcoords, zcoords) to make it consistent.

In your use case, the code runs without errors because the BEV grid is assumed to be symmetric (100x100), but the code fails when the BEV grid is asymmetric.

Based on the config file of the checkpoints released, I am trying to understand whether the model was pre-trained using a slightly different configuration or a different pre-processing of the dataset.
More specifically, what does the following mean?
"pretrained_model": "pyrocctrandetr_s_0904_100x100_gt100_200down100up_dla64_dice_adam_lr5e5_di3",
"load_ckpt": "checkpoint-0032.pth.gz"

Hi, thanks for your great work!
When I read your code, I am confused about the use of calib in class sample_polar2cartin in bev_transformer.py.

Thanks for your amazing work!

Do you have any plan to release your checkpoint trained on the whole NuScenes dataset?

Traceback (most recent call last):
File "D:/BEV/translating-images-into-maps-main/translating-images-into-maps-main/train.py", line 951, in
main()
File "D:/BEV/translating-images-into-maps-main/translating-images-into-maps-main/train.py", line 938, in main
train(args, train_loader, model, optimizer, epoch)
File "D:/BEV/translating-images-into-maps-main/translating-images-into-maps-main/train.py", line 53, in train
for i, ((image, calib, grid2d), (cls_map, vis_mask)) in enumerate(dataloader):
File "E:\ananconda\lib\site-packages\torch\utils\data\dataloader.py", line 359, in iter
return self._get_iterator()
File "E:\ananconda\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "E:\ananconda\lib\site-packages\torch\utils\data\dataloader.py", line 918, in init
w.start()
File "E:\ananconda\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "E:\ananconda\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "E:\ananconda\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "E:\ananconda\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "E:\ananconda\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Environment' object
Traceback (most recent call last):
File "", line 1, in
File "E:\ananconda\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "E:\ananconda\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

how to slove this problem

Hi @avishkarsaha , thank you for the great work.

I am trying to run inference with your pretrained weights, however am having trouble.

The pretrained weights seem to be using (according to config.txt) PyrOccNet_S_0914_old_rep100x100_out100x100.

However, this model does not exist in network.py. Can you please share the updated network.py? And give more guidance on how to infer and visualize? Thank you!

When I try using the default model_name PyrOccTranDetr_S_0904_old_rep100x100_out100x100, there is a huge mismatch between available keys

The resulting predictions does not look representative of the results

Traceback (most recent call last):
File "D:\bishe\translating-images-into-maps-main\train.py", line 945, in
main()
File "D:\bishe\translating-images-into-maps-main\train.py", line 932, in main
train(args, train_loader, model, optimizer, epoch)
File "D:\bishe\translating-images-into-maps-main\train.py", line 53, in train
for i, ((image, calib, grid2d), (cls_map, vis_mask)) in enumerate(dataloader):
File "D:\Download\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 441, in iter
return self._get_iterator()
File "D:\Download\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 388, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "D:\Download\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1042, in init
w.start()
File "D:\Download\envs\pytorch\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "D:\Download\envs\pytorch\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\Download\envs\pytorch\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "D:\Download\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "D:\Download\envs\pytorch\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Environment' object
Traceback (most recent call last):
File "", line 1, in
File "D:\Download\envs\pytorch\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Download\envs\pytorch\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

how to slove this problem

Dear @avishkarsaha,

thanks for the awesome work and sharing the code!

Is it possible that you also share the visualization code? There may be some files missed under ./src, such as "src.visualization".

I am highly interested in the performance of large-scale environments.

Thank you!

Bests,

Haoming

Hi, thanks for your nice work.

Can you tell me whether the invisible vehicle is masked when your model is compared with LSS and FIERY?

Best wishes.

loading val data
Traceback (most recent call last):
File "train.py", line 945, in
main()
File "train.py", line 860, in main
n_dec_layers=args.n_dec_layers,
File "/home/csib/Y_C/translating-images-into-maps/src/model/network.py", line 1009, in init
backbone_name=frontend, pretrained=pretrained
File "/home/csib/Y_C/translating-images-into-maps/src/model/backbone_utils.py", line 90, in resnet_fpn_backbone
pretrained=pretrained, norm_layer=misc_nn_ops.FrozenBatchNorm2d
File "/home/csib/Y_C/translating-images-into-maps/src/model/resnet.py", line 841, in resnet50
return _resnet("resnet50", Bottleneck, [3, 4, 6, 3], pretrained, progress, **kwargs)
File "/home/csib/Y_C/translating-images-into-maps/src/model/resnet.py", line 793, in _resnet
state_dict = load_state_dict_from_url(model_urls[arch], progress=progress)
File "/home/csib/anaconda3/envs/bev37/lib/python3.7/site-packages/torch/hub.py", line 731, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location)
File "/home/csib/anaconda3/envs/bev37/lib/python3.7/site-packages/torch/serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/csib/anaconda3/envs/bev37/lib/python3.7/site-packages/torch/serialization.py", line 920, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'.

Hi,

Is there a specific reason why only the first vector out of the nn.Embedding is ever used? tgt8 ... tgt64 are always zeros at this stage so you end up picking the 0-th vector for each spatial position, or in other words qe8 .. qe64 will always be filled with identical repeating values.... the values in the Embedding will obviously change over time with training but they will always be repeated throughout the qe-s

After permuting qe8 to [batch, spatial, spatial, len_embedding]:

tensor([[[ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         ...,
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921]],

        [[ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         ...,
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921]],

        [[ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         ...,
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921],
         [ 0.8230, -1.4024, -0.8630,  ..., -1.0915, -0.6329,  0.0921]]
....

I could not download nuscenes datasets from the url "https://www.icloud.com/iclouddrive/0aaSjW59DEqgUDKyy1uw0iSVg#nuscenes%5Fdata" due to readable-only. I want to know how to get it. Thanks!

Hi!

First of all, thanks for the excellent work and for sharing the code, looks really promising!

However I am having a hard time setting up the environment, getting all sorts of conflicts....would it be possible to have requirements file for conda or pip? That would make things much more easy.

Thanks again and kind regards,
Gianluca

Hi,
I downloaded nuscenes_data from the drive, put it the "translating-images-into-..." file. But I still have this issue (see picture). Where is the problem, please ?

Hello！Thanks for your good work ! Could you provide data.imdb?

Let me ask if train model with full nuscenes data, what is your configuration need, I have problem with too much RAM when training, my RAM is 13GB, I use colab or kaggle

Thank you very much

Thank you for sharing! I have some questions about the code:

1. When I run the train.py, this error occurred:
   lmdb.Error: /home/###/Downloads/translating-images-into-maps-main/nuscenes_data/lmdb/samples/CAM_FRONT: No such file or directory, It is about the lmdb.open. I guess it is caused by missing data.mdb in /nuscene_data/samples/CAM_FRONT.

2. where is the verification file? Files like validation.py

3. How to train the bigger Dataset, something like v1.0-trainval or others? Are there any Readme about dataloader?

Thanks again for sharing! Looking forward to your reply.

Thanks for the great work! The results look very interesting and I'd love to visualize the prediction outputs of the model on unseen data.

1. Can you release the pre-trained model that was used to get the results in the paper some time soon?
2. How would you produce the example visualizations shown in the paper after passing images through the model?

Thanks again.

Sorry, can you provide pre-train model with the same results as the paper?
Because my own train results are very poor

I ran the pre-trained model on the nuScenes mini dataset.

However, the model's predicted results appear to be collapsing.
Does anyone have the same phenomenon or the correct mask predicted?

I am using checkpoint-0020.pht.gz.
Fine-tuning is not done.

I downloaded the nuscenes_data from the google drive and upon running train.py, it ends in this error.

loading train data
Traceback (most recent call last):
  File "/N/projects/translating-images-into-maps/train.py", line 945, in <module>
    main()
  File "/N/projects/translating-images-into-maps/train.py", line 804, in main
    train_data = nuScenesMaps(
  File "/N/projects/translating-images-into-maps/src/data/dataloader.py", line 87, in __init__
    self.images_db = lmdb.open(
lmdb.Error: /N/projects/translating-images-into-maps/nuscenes_data/lmdb/samples/CAM_FRONT: No such file or directory

In the Google drive and subsequently in the downloaded files, data.mbd is missing

ls -latr /N/projects/translating-images-into-maps/nuscenes_data/lmdb/samples/CAM_FRONT
total 24
-rw-rw-r--@ 1 deepakduggirala  staff  8192 Jun 13 12:02 lock.mdb
-rw-rw-r--@ 1 deepakduggirala  staff   162 Jun 13 12:02 .data.mdb.icloud
drwxr-xr-x@ 4 deepakduggirala  staff   128 Nov 21 00:23 .
drwxr-xr-x@ 4 deepakduggirala  staff   128 Nov 21 00:24 ..

Could you please re-upload the data.mdb file to google drive?

lmdb.InvalidError: /mnt/Disk16T/meh/translating-images-into-maps/nuscenes_data/lmdb/samples/CAM_FRONT: MDB_INVALID: File is not an LMDB file。

您在谷歌云盘提供的数据集有些是.icloud后缀，失效。可以重新上传吗？

I am unable to download your pretrained model. Thank you.

Can you please republish the data that you have used as it is no longer available?

Hi there,

Very interesting research, thanks for sharing your work!

I was wondering when will the checkpoints used to output the results shown in your paper be released? You mentioned a release quite a while ago but I couldn't find it on your iCloud repository. It takes some time to train on the whole dataset, so it would help a lot to start from something already trained.

Thanks a lot!

Quentin

Hi. Thanks for the amazing work and contribution!
I wonder why the batch size and learning rate are set differently in the source code from the paper.
For faster learning and less memory occupation with a smaller dataset?

The paper stated:
"We train our network end-to-end with an Adam
optimizer, batch size 8 and initial learning rate of 5e−5,
which we decay by 0.99 every epoch for 40 epochs"

But in the source code:
parser.add_argument("-l", "--lr", type=float, default=1e-4, help="learning rate")
and
parser.add_argument( "-b", "--batch-size", type=int, default=1, help="mini-batch size for training" )
Thanks!

Thank you for sharing code! I'm very interested in your excellent work! I I'd like to ask you some questions.
I visualized the nuscenes dataset (first image), but it contains lots of noise in black regions and different from the one (second image) in your paper.
In my understanding, black region is invisible regions of lidar (i.e., "~lidar_ray_mask" in your code?)
Could you tell me how to reduce these noises in black regions?
I look forward to your reply very much!!!

Dear @avishkarsaha ,

First of all, thanks for the excellent work and sharing the code!

It was easy to set up the environment and train the model.

I am interested in testing the model on new images and check/compare the results. How can i add my own images to the trained model? Any kind of information about the dimensions and type of the images used for the model will be useful.

Thank you!

John

Excellent work. Please release the full code. Thank you.

Excuse me, after I download the code and configure the environment, I will be prompted with a ValueError: Invalid format string error in line 732 during the operation. How can I fix it?

Hi,Thank you for your contribution to this work.
When I run train.py, encountered a ValueError: only one element tensors can be converted to Python scalars in line 303, in compute_ loss
label = torch. tensor(labels, dtype=torch.long)
I have tried to convert it into label = torch.stack (labels) or convert list into array and then into tensor, but none of them works.
How can I solve this problem?

Hi, thank you for sharing code!
I'm very insterested in this great work. When I read the code, I am confused about dataset configuration:

1. It seems you adopt data only from CAM_FRONT when training with mini nuscenes dataset, is it the same when using full nuscenes dataset?
2. In code of https://github.com/tom-roddick/mono-semantic-maps, they contain all 6 cameras to generate monocular data. Do you implement their code in your dataset when doing comparation?

Looking forward to your reply, thanks in advance!

We are studying TIIM and found many problems, the most critical confusions are:

1. This codes employs model/transformer/Transformer but not the model/transformer/TransformerMonotonic, which should be the main ideal of using MoCha mentioned in the paper.
2. In TransformerMonotonic, the image features are organized in HxNWxC, that means features are scanned in the order of row by row, not column by column, but the paper puts emphasis on column and explained why vertical features do better to the translating.

Did I misunderstand the whole thing in paper and codes? Please correct me if I'm wrong.

Thanks!

Hi, thanks for your fantastic work.

As mentioned in issue #7 you were going to release the checkpoint trained on the whole Nuscenes dataset. I am writing to request this checkpoint which is important for my recent work. I have tried to train the model on the mini dataset you provided and evaluated it on new images (with close domain with Nuscenes data). However, I got terrible results. Therefore I would like to test the model trained on the whole dataset on these images. (The whole Nuscenes dataset is too large for me to train.)

Thank you again for your work.

Best wishes

how to generate nuscenes dataset