nightmare-n / unipad Goto Github PK
View Code? Open in Web Editor NEWUniPAD: A Universal Pre-training Paradigm for Autonomous Driving (CVPR 2024)
Home Page: https://arxiv.org/abs/2310.08370
License: Apache License 2.0
UniPAD: A Universal Pre-training Paradigm for Autonomous Driving (CVPR 2024)
Home Page: https://arxiv.org/abs/2310.08370
License: Apache License 2.0
when i debug with you code, found that there is no pts_voxel_encoder, thus there is no pts_feat of the output.
I can only found you use point surpervise the sdf in "sparse_points_sdf_supervised"?
So how do you use lidar input like your paper?
Can I use this project to train a 3D backbone purely on lidar? Because I dont have corresponding image data
Thank you for releasing this amazing work! I just had a couple questions on some of the camera-only outdoor details @Nightmare-n
Apologies for the list of questions, but I'm really interested in the work. Again, thank you so much in advance!
Thank you for your meaningful work. How to get merged_uvtr_fusion_vs0.075_finetune.pth
Hi, thank you for your amazing work.
As indicated in your project config files, the weights for finetuning fusion-based model are got by "merging the weights of uvtr_lidar and uvtr_cam". So, from that statement, uvtr_lidar and uvtr_cam (pretraining phase) are trained individually before, and you just combine state_dicts of those 2 pretrained models to use for finetuning fusion-based model.
But when I first read your paper, I thought the fusion weights are got by training both lidar and camera branches "simultaneously" (as indicated in your framework), which is quite different from the above implementation.
This makes me confused and I want to ask you if I am understanding your ideas correctly.
Is "camera-based pretrained weights + lidar-based pretrained weights = fusion-based pretrained weights" what you mean in your paper too?
Thank you so much in advance.
Hello, thanks for your excellent work! I have downloaded nuScenes dataset. Can you provide an explanation of running test data?
I observed point_nsample==512 in the code. I would like to ask the author whether he has done ablation experiments on point_nsample. According to my understanding, the larger the point_nsample, the more supervision signals, and the effect should be better.
Hi, thanks for this great work.
I wanna know how to get rendered RGB&D-images as you've shown in your paper, could you plz give me some advice?
Please ask the author if he has verified the effectiveness of the Unipad framework on other backbones.
For example resnet-50. If so, what can I do to verify that resnet-50 is pre-trained under unipad framework.
Hello,
Thank you for your excellent work!
There is a bug in the test of UVTRSSL:
Hello, thanks for your excellent work! I found your method concise and elegant, and these days I'm trying to replicate it. I have a few questions about the code.
1. It seems that the provided code only contains camera modality and the lidar modality is missing. Could you please provide the config files of UniPAD that use multi-modality and lidar-only UVTR? It would be very helpful.
2. I ran the code successfully, but I found the result metric was lower than that in the paper. I ran the project in many ways, all of them are based on the complete nuScenes dataset and your configs:
a. I pretrained and finetuned the model from scratch.
b. From the provided Google drive link, I downloaded your official pretrained pth file, and finetuned the model based on it.
c. I downloaded your official pretrained and finetuned pth files, and directly test the model based on your finetuned pth.
Sadly, the result mAP metrics of the 3 ways are all approximately 32. Due to the camera-only setting in the code, I think the corresponding mAP metric in the paper should be UVTR-C + UniPAD, which is 41.5. So I wonder did I miss something critical?
Looking forward to your early reply, and thanks in advance!
Hi, thank you for your great work!
One simple question, during pre-training, do you use full nuScene training data? For downstream task, 3DOD as example, do you still train on the same data used by pre-training?
A very much appreciation to your work!
Could you please provide your timeline for releasing your code?
Thank you for your work. I would like to ask how sampling 512 rays per image view achieves such a high rendering resolution in the final render image(figure3 in your paper). For a high-definition image of 1600 900, 512 seems like a very small number.
Hello,
Thanks for the great work, and for making the code available.
In the arxiv paper, you show results with:
[EDIT]:
Could it be that there is a transpose in the table, and that results should be:
Hi. Thank you for your great work.
May I ask you some questions regarding how you get the rendering results in Figure 3 of your published paper?
Did you use the frozen pretrained unipad-c (without masking the input images) to render both rgb and depth images? And here, the number of rays to sample is corresponding to the image resolution (like 928x1600 rays per each camera view or 928 // k x 1600 //k rays,....,?
Thank you.
Hi. Thank you for your interesting work.
I have a question about the ablation study of your paper in Table 5.
Did you mean you only take the view transformation from BEVDet, BEVDepth and BEVFormer while keeping other components like image backbone, detection head, ... same as in UVTR-cam?
Could you please provide the config files for these studies?
Thank you in advance.
Hi team, thanks for releasing this exceptional work.
In the released log (abl_uvtr_cam_vs0.1_finetune.log), a pretrained model can be observed in Line 1236
2023-11-18 18:34:11,643 - mmdet - INFO - load checkpoint from work_dirs/convnext_s_pretrain_enorm_nuscenes_fp32_d32_x1_nofar/epoch_12.pth
Does this indicate that the UniPAD model is pretrained with FP32?
Also, we notice that the default pretrain setting in this repo is FP16 (fp16_enabled=True). Could you share the comparison between FP32 pretraining and FP16 pretraining?
Thanks again for your attention, and look forward to your reply.
are there Pointcloud_Backbone? Only img_backbone?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.