Comments (8)
By the way, I'm using the following command to evaluate on val split:
python eval.py --dataset_directory ldif/data/ShapeNet/ldif_dataset/ --experiment_name {example} --split val --use_inference_kernel --result_directory {example} --save_ldifs --save_results --save_meshes --eval_frac 0.1
It is worth mentioning that param 'eval_frac' is used. Don't know if this has anything to do with the behavior.
A ShapeNet model and its processed watertight mesh look like below:
from ldif.
Update: I tried a full test on val split. The result is the same: IoU=81.81, Chamfer=0.045, F-Score=90.38.
from ldif.
Hi Cheng, thanks very much for bringing this to our attention.
For display purposes the chamfer distance of all methods is scaled up (equation in the appendix). The code currently scales it up by a factor of 100, but looking at the output logs for the paper, it looks like we really used a scale of 1000; this is a mistake. Any chamfer you have previously computed can be multiplied by 10x. I will push a patch that increases the scale from 100x to 1000x in ldif/inference/metrics.py.
from ldif.
Commit 3482786 should fix this.
As an aside, the F-Score and IoU at that stage seem fairly expected to me. The IoU and F-Score should continue to climb slowly until training completes. To be honest, training never really 'completes'- even the LDIF models used for the paper were continuing to improve (albeit very slowly) on the val set, we just stopped training at some point. I have not seen an LDIF on ShapeNet fully converge. SIF converges much more quickly though and val performance flattens out.
from ldif.
Thanks for your quick fix and the kind reminder about the training process!
However, the training speed is not so ideal on a 1080Ti GPU. The 200k iterations' training took me more than 4 days to complete. It looks like that the bottleneck is on the reading speed of my hard disk which I can't improve easily.
May I ask what kind of hardware did you train on and how long have you taken to complete the 1M iterations' training?
Furthermore, may I ask when or if will the checkpoint be released?
from ldif.
Hmm, that is several times slower than the performance on the hardware I tested on (a V100 with the data on an SSD). The IO code was newly written for the open source release. For the paper we trained on a cluster and had a more optimized IO pipeline, where more of the processing was done before training rather than on-the-fly.
Given that the input process code is simpler than the previous pipeline and does a nontrivial amount of work, it is not too surprising it can bottleneck on some hardware. I think I can resolve this issue, but it will take some time.
In order to help make sure your issue is addressed, can you tell me whether the code appears to be bottlenecked on IO throughput (the drive is bottlenecked filling read requests), IO latency (there aren't enough concurrent read requests to keep either the drive or CPU busy), or CPU performance (the CPU is at 100%)? My guess is that it is IO throughput, but since it is not occurring on my machine I can't be sure.
I can share a more fully trained checkpoint with you directly, if you would like, but the original checkpoints that were used to write the paper are incompatible with this code base (they have dependencies that could not be open-sourced). I think the long-term fix to this problem is probably multi-gpu support + a more optimized IO pipeline (like tf-records coupled with less data read per example)
I also updated #2 regarding the checkpoint release.
from ldif.
CPU usage is around 20% on a 16 core CPU i7-9800X. GPU is not fully running with usage like below:
The data is stored on a central server with a GbE network shared by several GPU servers. So it is hard to know the exact IO latency. IO throughput is around 80MB/s seen from the network usage, which should be the bottleneck. Your guess should be quite accurate.
I'll try some improvement on the storage bottleneck. At the same time, a fully trained checkpoint would be great. How can I get in touch with you personally? Thank you for your patience by the way!
from ldif.
Update: I'm trying to get in touch with you personally though your princeton.edu email. Thanks in advance!
from ldif.
Related Issues (13)
- Fix issues while running ./build_gaps.sh HOT 3
- /process_mesh_local.sh: line 47: 13856 Bus error (core dumped) ${gaps}/msh2msh $mesh_in $mesh -scale_by_pca -translate_by_centroid -scale 0\.25 -debug_matrix ${outdir}/orig_to_gaps.txt
- Segmentation fault when train the net HOT 3
- Is it possible to share your preprocessed data and pretrained model of LDIF? HOT 4
- Visualization with qview: scaling issue
- no limitation for "%s" while calling fscanf()
- Error when unit testing
- The result of unit_test.sh HOT 1
- IO Performance Issues HOT 2
- Crash on train HOT 1
- test or val issue HOT 2
- Error in data preprocessing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ldif.