Comments (10)
Hi @mertkiray,
This is actually expected.
When trying the processing code of Total3D, we find that the processed results have fewer samples than the downloaded pre-processed data from Total3D.
Since the pre-processing code from Total3D filters out some of the scenes based on several rules, we suspect that their code might have some change after the release which caused more scenes to be ignored.
Since our pre-processing code is based on Total3D's released code, we inherit their train/test set of the dataset processed with it.
The missing samples should not cause a large difference in the evaluation results.
from implicit3dunderstanding.
Hi @chengzhag,
Thank you for the detailed answer. I am asking because the model I trained seems to differ on 3D Object detection results from the paper.
Top row is from the paper while the bottom row is from the output of training.
Also I have similar issues with the visualizations differing from the visualizations from paper results.
I wanted to make sure this is not because of preprocessing. Do you have any idea why these results are different as well as visualizations?
Also how can I get other test results you provided in the paper like Layout IoU, cam pitch, cam roll and chamfer distance.
Thank you so much 😊
from implicit3dunderstanding.
Hi @mertkiray,
The LDIF does seem to break on some of the objects with thin structures, especially when generalizing from Pix3D to SUN RGB-D.
The other metrics can be checked after the training from the log file in the experiment folder or can be better shown from wandb experiment page (you can create a report with tables or graphs like below)
Would you please check the training and testing curve to see if there is any problem?
from implicit3dunderstanding.
Thank you @chengzhag for sharing other metrics,
I am sharing my training and testing curves below.
This is LEN training/testing curves :
Do you see any problems with this curves?
from implicit3dunderstanding.
Hi, @mertkiray,
I presume that the rest of the metrics logged with wandb are OK?
I see that your curves are quite similar to what I have got.
From my observation, It does require some luck (or several attempts) to achieve the results of our provided checkpoint, since there might be some randomness during the training process and the checkpoint we provided is the best we can get.
from implicit3dunderstanding.
In LEN logs:
Test loss (layout_iou): 0.620150 Test loss (cam_pitch_err): 3.728792 Test loss (cam_roll_err): 2.606081
IN LIEN+LDIF logs:
Test loss (Avg_Chamfer): 0.008465 {'bed': 0.005958032906252605, 'wardrobe': 0.0031370156852062792, 'bookcase': 0.004260495355346292, 'chair': 0.007310174998500854, 'sofa': 0.003096780832186188, 'table': 0.017010838347444775, 'desk': 0.013083522971386888, 'misc': 0.02592532049439957, 'tool': 0.0029951145195148206} Test loss (chamfer_woICP): 8.669688 {'bed': 6.097681875626751, 'wardrobe': 3.2926102414548657, 'bookcase': 4.573465430221525, 'chair': 7.50823779989532, 'sofa': 3.34066853316409, 'table': 17.324360070516153, 'desk': 12.981401550831366, 'misc': 26.215719928701777, 'tool': 3.0616522062591085} Test loss (chamfer_wICP): 7.467241 {'bed': 4.969611945433475, 'wardrobe': 3.026142153369496, 'bookcase': 3.9906250437764443, 'chair': 6.427196312469815, 'sofa': 3.6129484184597493, 'table': 14.570997367973707, 'desk': 10.58210799721211, 'misc': 22.70249547140817, 'tool': 2.7223403830129893}
In SGCN logs:
Test loss (layout_iou): 0.637629 Test loss (cam_pitch_err): 3.061840 Test loss (cam_roll_err): 2.257464 Test loss (iou_3d): 0.198359 Test loss (iou_2d): 0.666520
In JOINT logs:
Test loss (layout_iou): 0.633157 Test loss (cam_pitch_err): 3.076085 Test loss (cam_roll_err): 2.250714 Test loss (iou_3d): 0.208100 Test loss (iou_2d): 0.676601 Test loss (Lg): 1.353826
I think they are close but still bad from the paper results. :) This is my first research paper that I tried to duplicate results and thank you for the amazing guidance :)
Do we have a chance to get the pretrained checkpoints for LEN and LIEN+LDIF?
from implicit3dunderstanding.
Hi @mertkiray
Your pretrained LEN is actually quite close to mine:
Test loss (layout_iou): 0.613996 Test loss (cam_pitch_err): 3.652064 Test loss (cam_roll_err): 2.517752
, which is trained with the parameters provided by Total3D.
Since Total3D already provides a pretrained model with similar testing results of LEN:
Test loss (layout_iou): 0.599689 Test loss (cam_pitch_err): 3.613924 Test loss (cam_roll_err): 2.455977
(even better on camera pose estimation, may be loaded as an alternative)
, there seems no need to provide another LEN pretrained model.
Since the LIEN+LDIF is not finetuned during the joint training, the weights in our provided final model should be identical to the pretrained weights.
You can load it to test the Chamfer loss.
from implicit3dunderstanding.
Hi @chengzhag thank you again :)
So there is nothing wrong I assume with my setup and the differences with the paper is just randomness?
from implicit3dunderstanding.
Hi @mertkiray,
I think yes.
On your demand, I also updated a link to download pretrained LIEN+LDIF model.
Hope this can help you.
Update: the link
from implicit3dunderstanding.
Thank you so much @chengzhag
from implicit3dunderstanding.
Related Issues (20)
- Error in building the project HOT 35
- confusion about the final chamfer distance for LDIF result HOT 3
- 2D detector output from Total3D HOT 2
- got error "nvcc fatal: Unsupported gpu architecture 'compute_30' " when run "python project.py build" HOT 1
- Error while running the demo via ssh HOT 2
- Error while running the test code HOT 1
- Error when training LDIF
- Error when I run the demo
- How to evaluate on ObjectNet3D
- Solving environment: failed for $ conda env create -f environment.yml HOT 2
- Pretraining weight link invalid HOT 1
- similar problem
- Errors about the pyTorchChamferDistance HOT 2
- the pretrained checkpoint of Total3DUnderstanding for ODN HOT 2
- About Layout Estimation Network Training HOT 2
- Testing typo HOT 1
- Visualization results different from paper outputs HOT 7
- Detection Evaluation Script in Python HOT 7
- librenderer.render() (glViewport) throws segfault HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from implicit3dunderstanding.