avishkarsaha / translating-images-into-maps Goto Github PK
View Code? Open in Web Editor NEWOfficial PyTorch code for 'Translating Images Into Maps' ICRA 2022 (Outstanding Paper Award)
License: Other
Official PyTorch code for 'Translating Images Into Maps' ICRA 2022 (Outstanding Paper Award)
License: Other
Dear @avishkarsaha,
thanks for the awesome work and sharing the code!
Is it possible that you also share the visualization code? There may be some files missed under ./src, such as "src.visualization".
I am highly interested in the performance of large-scale environments.
Thank you!
Bests,
Haoming
Hi,Thank you for your contribution to this work.
When I run train.py, encountered a ValueError: only one element tensors can be converted to Python scalars in line 303, in compute_ loss
label = torch. tensor(labels, dtype=torch.long)
I have tried to convert it into label = torch.stack (labels) or convert list into array and then into tensor, but none of them works.
How can I solve this problem?
Traceback (most recent call last):
File "D:/BEV/translating-images-into-maps-main/translating-images-into-maps-main/train.py", line 951, in
main()
File "D:/BEV/translating-images-into-maps-main/translating-images-into-maps-main/train.py", line 938, in main
train(args, train_loader, model, optimizer, epoch)
File "D:/BEV/translating-images-into-maps-main/translating-images-into-maps-main/train.py", line 53, in train
for i, ((image, calib, grid2d), (cls_map, vis_mask)) in enumerate(dataloader):
File "E:\ananconda\lib\site-packages\torch\utils\data\dataloader.py", line 359, in iter
return self._get_iterator()
File "E:\ananconda\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "E:\ananconda\lib\site-packages\torch\utils\data\dataloader.py", line 918, in init
w.start()
File "E:\ananconda\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "E:\ananconda\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "E:\ananconda\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "E:\ananconda\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "E:\ananconda\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Environment' object
Traceback (most recent call last):
File "", line 1, in
File "E:\ananconda\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "E:\ananconda\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
how to slove this problem
Traceback (most recent call last):
File "D:\bishe\translating-images-into-maps-main\train.py", line 945, in
main()
File "D:\bishe\translating-images-into-maps-main\train.py", line 932, in main
train(args, train_loader, model, optimizer, epoch)
File "D:\bishe\translating-images-into-maps-main\train.py", line 53, in train
for i, ((image, calib, grid2d), (cls_map, vis_mask)) in enumerate(dataloader):
File "D:\Download\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 441, in iter
return self._get_iterator()
File "D:\Download\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 388, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "D:\Download\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1042, in init
w.start()
File "D:\Download\envs\pytorch\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "D:\Download\envs\pytorch\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\Download\envs\pytorch\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "D:\Download\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "D:\Download\envs\pytorch\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Environment' object
Traceback (most recent call last):
File "", line 1, in
File "D:\Download\envs\pytorch\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Download\envs\pytorch\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
how to slove this problem
Hi,
Is there a specific reason why only the first vector out of the nn.Embedding is ever used? tgt8
... tgt64
are always zeros at this stage so you end up picking the 0-th vector for each spatial position, or in other words qe8
.. qe64
will always be filled with identical repeating values.... the values in the Embedding will obviously change over time with training but they will always be repeated throughout the qe
-s
translating-images-into-maps/src/model/network.py
Line 1150 in 92b9627
translating-images-into-maps/src/model/network.py
Line 1161 in 92b9627
After permuting qe8
to [batch, spatial, spatial, len_embedding]:
tensor([[[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
...,
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921]],
[[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
...,
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921]],
[[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
...,
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921],
[ 0.8230, -1.4024, -0.8630, ..., -1.0915, -0.6329, 0.0921]]
....
Hi @avishkarsaha ,
translating-images-into-maps/src/utils.py
Line 1316 in 92b9627
I think there is a bug in the above referenced line of the make_grid2d() function that you invoke in the dataloader. For the grid to be consistent with the tensor slicing implemented here, I believe that the order of zz and xx should be reversed. In short, zz, xx = torch.meshgrid(zcoords, xcoords)
should be replaced with xx, zz = torch.meshgrid(xcoords, zcoords)
to make it consistent.
In your use case, the code runs without errors because the BEV grid is assumed to be symmetric (100x100), but the code fails when the BEV grid is asymmetric.
I could not download nuscenes datasets from the url "https://www.icloud.com/iclouddrive/0aaSjW59DEqgUDKyy1uw0iSVg#nuscenes%5Fdata" due to readable-only. I want to know how to get it. Thanks!
Thanks for your amazing work!
Do you have any plan to release your checkpoint trained on the whole NuScenes dataset?
Sorry, can you provide pre-train model with the same results as the paper?
Because my own train results are very poor
Thank you for sharing code! I'm very interested in your excellent work! I I'd like to ask you some questions.
I visualized the nuscenes dataset (first image), but it contains lots of noise in black regions and different from the one (second image) in your paper.
In my understanding, black region is invisible regions of lidar (i.e., "~lidar_ray_mask" in your code?)
Could you tell me how to reduce these noises in black regions?
I look forward to your reply very much!!!
Hi @avishkarsaha , thank you for the great work.
I am trying to run inference with your pretrained weights, however am having trouble.
The pretrained weights seem to be using (according to config.txt
) PyrOccNet_S_0914_old_rep100x100_out100x100
.
However, this model does not exist in network.py
. Can you please share the updated network.py
? And give more guidance on how to infer and visualize? Thank you!
When I try using the default model_name PyrOccTranDetr_S_0904_old_rep100x100_out100x100
, there is a huge mismatch between available keys
The resulting predictions does not look representative of the results
Hi, thanks for your fantastic work.
As mentioned in issue #7 you were going to release the checkpoint trained on the whole Nuscenes dataset. I am writing to request this checkpoint which is important for my recent work. I have tried to train the model on the mini dataset you provided and evaluated it on new images (with close domain with Nuscenes data). However, I got terrible results. Therefore I would like to test the model trained on the whole dataset on these images. (The whole Nuscenes dataset is too large for me to train.)
Thank you again for your work.
Best wishes
Excellent work. Please release the full code. Thank you.
Can you please republish the data that you have used as it is no longer available?
loading val data
Traceback (most recent call last):
File "train.py", line 945, in
main()
File "train.py", line 860, in main
n_dec_layers=args.n_dec_layers,
File "/home/csib/Y_C/translating-images-into-maps/src/model/network.py", line 1009, in init
backbone_name=frontend, pretrained=pretrained
File "/home/csib/Y_C/translating-images-into-maps/src/model/backbone_utils.py", line 90, in resnet_fpn_backbone
pretrained=pretrained, norm_layer=misc_nn_ops.FrozenBatchNorm2d
File "/home/csib/Y_C/translating-images-into-maps/src/model/resnet.py", line 841, in resnet50
return _resnet("resnet50", Bottleneck, [3, 4, 6, 3], pretrained, progress, **kwargs)
File "/home/csib/Y_C/translating-images-into-maps/src/model/resnet.py", line 793, in _resnet
state_dict = load_state_dict_from_url(model_urls[arch], progress=progress)
File "/home/csib/anaconda3/envs/bev37/lib/python3.7/site-packages/torch/hub.py", line 731, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location)
File "/home/csib/anaconda3/envs/bev37/lib/python3.7/site-packages/torch/serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/csib/anaconda3/envs/bev37/lib/python3.7/site-packages/torch/serialization.py", line 920, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'.
I downloaded the nuscenes_data from the google drive and upon running train.py, it ends in this error.
loading train data
Traceback (most recent call last):
File "/N/projects/translating-images-into-maps/train.py", line 945, in <module>
main()
File "/N/projects/translating-images-into-maps/train.py", line 804, in main
train_data = nuScenesMaps(
File "/N/projects/translating-images-into-maps/src/data/dataloader.py", line 87, in __init__
self.images_db = lmdb.open(
lmdb.Error: /N/projects/translating-images-into-maps/nuscenes_data/lmdb/samples/CAM_FRONT: No such file or directory
In the Google drive and subsequently in the downloaded files, data.mbd is missing
ls -latr /N/projects/translating-images-into-maps/nuscenes_data/lmdb/samples/CAM_FRONT
total 24
-rw-rw-r--@ 1 deepakduggirala staff 8192 Jun 13 12:02 lock.mdb
-rw-rw-r--@ 1 deepakduggirala staff 162 Jun 13 12:02 .data.mdb.icloud
drwxr-xr-x@ 4 deepakduggirala staff 128 Nov 21 00:23 .
drwxr-xr-x@ 4 deepakduggirala staff 128 Nov 21 00:24 ..
Could you please re-upload the data.mdb file to google drive?
Based on the config file of the checkpoints released, I am trying to understand whether the model was pre-trained using a slightly different configuration or a different pre-processing of the dataset.
More specifically, what does the following mean?
"pretrained_model": "pyrocctrandetr_s_0904_100x100_gt100_200down100up_dla64_dice_adam_lr5e5_di3",
"load_ckpt": "checkpoint-0032.pth.gz"
Hi there,
Very interesting research, thanks for sharing your work!
I was wondering when will the checkpoints used to output the results shown in your paper be released? You mentioned a release quite a while ago but I couldn't find it on your iCloud repository. It takes some time to train on the whole dataset, so it would help a lot to start from something already trained.
Thanks a lot!
Quentin
lmdb.InvalidError: /mnt/Disk16T/meh/translating-images-into-maps/nuscenes_data/lmdb/samples/CAM_FRONT: MDB_INVALID: File is not an LMDB file。
您在谷歌云盘提供的数据集有些是.icloud后缀,失效。可以重新上传吗?
Hi, thanks for your great work!
When I read your code, I am confused about the use of calib in class sample_polar2cartin
in bev_transformer.py.
x_zmax = self.z_max / f * (u_max - cu)
Excuse me, after I download the code and configure the environment, I will be prompted with a ValueError: Invalid format string error in line 732 during the operation. How can I fix it?
We are studying TIIM and found many problems, the most critical confusions are:
Did I misunderstand the whole thing in paper and codes? Please correct me if I'm wrong.
Thanks!
Dear @avishkarsaha ,
First of all, thanks for the excellent work and sharing the code!
It was easy to set up the environment and train the model.
I am interested in testing the model on new images and check/compare the results. How can i add my own images to the trained model? Any kind of information about the dimensions and type of the images used for the model will be useful.
Thank you!
John
Hi, thank you for sharing code!
I'm very insterested in this great work. When I read the code, I am confused about dataset configuration:
Looking forward to your reply, thanks in advance!
Thanks for the great work! The results look very interesting and I'd love to visualize the prediction outputs of the model on unseen data.
Thanks again.
Hello!Thanks for your good work ! Could you provide data.imdb?
Hello!Thanks for your good work ! Could you please explain what GPU you used to train models, and how long does it take?
I am unable to download your pretrained model. Thank you.
Let me ask if train model with full nuscenes data, what is your configuration need, I have problem with too much RAM when training, my RAM is 13GB, I use colab or kaggle
Thank you very much
I downloaded the source code and nuscene data(mini version, provided by the Author) as README.txt told, after training 1000 epochs:
whatever the value of data_size is, any input image will get an nonsense output. I tried several different images(from nuscene or mine) but the result images look similar. I have no ideal where is the problem.
So have anyone succeeded in training a model to get a good result?
Hi. Thanks for the amazing work and contribution!
I wonder why the batch size and learning rate are set differently in the source code from the paper.
For faster learning and less memory occupation with a smaller dataset?
The paper stated:
"We train our network end-to-end with an Adam
optimizer, batch size 8 and initial learning rate of 5e−5,
which we decay by 0.99 every epoch for 40 epochs"
But in the source code:
parser.add_argument("-l", "--lr", type=float, default=1e-4, help="learning rate")
and
parser.add_argument( "-b", "--batch-size", type=int, default=1, help="mini-batch size for training" )
Thanks!
how to generate nuscenes dataset
Hi!
First of all, thanks for the excellent work and for sharing the code, looks really promising!
However I am having a hard time setting up the environment, getting all sorts of conflicts....would it be possible to have requirements file for conda or pip? That would make things much more easy.
Thanks again and kind regards,
Gianluca
Hi, thanks for your nice work.
Can you tell me whether the invisible vehicle is masked when your model is compared with LSS and FIERY?
Best wishes.
Thank you for sharing! I have some questions about the code:
When I run the train.py, this error occurred:
lmdb.Error: /home/###/Downloads/translating-images-into-maps-main/nuscenes_data/lmdb/samples/CAM_FRONT: No such file or directory, It is about the lmdb.open. I guess it is caused by missing data.mdb in /nuscene_data/samples/CAM_FRONT.
where is the verification file? Files like validation.py
How to train the bigger Dataset, something like v1.0-trainval or others? Are there any Readme about dataloader?
Thanks again for sharing! Looking forward to your reply.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.