Comments (7)
VictorZoo
commented on May 20, 2024
Some details are as follows:
7Scenes
with author's model
Evaluate scene chess: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_chess.txt
[12/06/2021 16:26:38 pixloc.utils.eval INFO]
Median errors: 0.028m, 0.956deg
Percentage of test images localized within:
1cm, 1deg : 5.10%
2cm, 2deg : 31.15%
3cm, 3deg : 55.80%
5cm, 5deg : 84.70%
25cm, 2deg : 90.25%
50cm, 5deg : 94.40%
500cm, 10deg : 94.85%
[12/06/2021 16:26:38 pixloc INFO] Evaluate scene fire: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_fire.txt
[12/06/2021 16:26:41 pixloc.utils.eval INFO]
Median errors: 0.024m, 0.954deg
Percentage of test images localized within:
1cm, 1deg : 9.10%
2cm, 2deg : 41.35%
3cm, 3deg : 63.15%
5cm, 5deg : 78.10%
25cm, 2deg : 79.15%
50cm, 5deg : 87.85%
500cm, 10deg : 92.85%
[12/06/2021 16:26:41 pixloc INFO] Evaluate scene heads: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_heads.txt
[12/06/2021 16:26:42 pixloc.utils.eval INFO]
Median errors: 0.011m, 0.836deg
Percentage of test images localized within:
1cm, 1deg : 39.30%
2cm, 2deg : 79.40%
3cm, 3deg : 84.30%
5cm, 5deg : 86.30%
25cm, 2deg : 82.80%
50cm, 5deg : 86.80%
500cm, 10deg : 87.80%
[12/06/2021 16:26:42 pixloc INFO] Evaluate scene office: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_office.txt
[12/06/2021 16:26:44 pixloc.utils.eval INFO]
Median errors: 0.035m, 1.049deg
Percentage of test images localized within:
1cm, 1deg : 3.25%
2cm, 2deg : 21.45%
3cm, 3deg : 39.92%
5cm, 5deg : 67.80%
25cm, 2deg : 81.20%
50cm, 5deg : 95.50%
500cm, 10deg : 97.10%
[12/06/2021 16:26:44 pixloc INFO] Evaluate scene pumpkin: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_pumpkin.txt
[12/06/2021 16:26:46 pixloc.utils.eval INFO]
Median errors: 0.052m, 1.463deg
Percentage of test images localized within:
1cm, 1deg : 1.85%
2cm, 2deg : 9.35%
3cm, 3deg : 21.05%
5cm, 5deg : 48.75%
25cm, 2deg : 61.15%
50cm, 5deg : 82.40%
500cm, 10deg : 84.95%
[12/06/2021 16:26:46 pixloc INFO] Evaluate scene redkitchen: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_redkitchen.txt
[12/06/2021 16:26:51 pixloc.utils.eval INFO]
Median errors: 0.045m, 1.482deg
Percentage of test images localized within:
1cm, 1deg : 1.62%
2cm, 2deg : 13.24%
3cm, 3deg : 30.08%
5cm, 5deg : 55.10%
25cm, 2deg : 65.08%
50cm, 5deg : 85.20%
500cm, 10deg : 88.78%
[12/06/2021 16:26:51 pixloc INFO] Evaluate scene stairs: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_7scenes_stairs.txt
[12/06/2021 16:26:52 pixloc.utils.eval INFO]
Median errors: 0.069m, 1.719deg
Percentage of test images localized within:
1cm, 1deg : 0.70%
2cm, 2deg : 8.20%
3cm, 3deg : 18.90%
5cm, 5deg : 36.60%
25cm, 2deg : 51.00%
50cm, 5deg : 74.00%
500cm, 10deg : 86.00%
pixloc_7scenes_chess.txt
pixloc_7scenes_fire.txt
pixloc_7scenes_heads.txt
pixloc_7scenes_office.txt
pixloc_7scenes_pumpkin.txt
pixloc_7scenes_redkitchen.txt
pixloc_7scenes_stairs.txt
from pixloc.
VictorZoo
commented on May 20, 2024
Cambridge
Evaluate scene OldHospital: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_Cambridge_OldHospital.txt
[12/06/2021 21:11:18 pixloc.utils.eval INFO]
Median errors: 0.508m, 0.831deg
Percentage of test images localized within:
1cm, 1deg : 0.00%
2cm, 2deg : 0.00%
3cm, 3deg : 0.00%
5cm, 5deg : 0.55%
25cm, 2deg : 31.32%
50cm, 5deg : 49.45%
500cm, 10deg : 94.51%
Evaluate scene StMarysChurch: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_Cambridge_StMarysChurch.txt
[12/06/2021 21:44:31 pixloc.utils.eval INFO]
Median errors: 0.143m, 0.354deg
Percentage of test images localized within:
1cm, 1deg : 0.38%
2cm, 2deg : 1.70%
3cm, 3deg : 4.15%
5cm, 5deg : 10.57%
25cm, 2deg : 70.19%
50cm, 5deg : 81.32%
500cm, 10deg : 86.60%
Evaluate scene ShopFacade: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_Cambridge_ShopFacade.txt
[12/06/2021 21:46:27 pixloc.utils.io INFO] Imported 103 images from pixloc_Cambridge_ShopFacade.txt
[12/06/2021 21:46:27 pixloc.utils.eval INFO]
Median errors: 0.055m, 0.218deg
Percentage of test images localized within:
1cm, 1deg : 0.00%
2cm, 2deg : 5.83%
3cm, 3deg : 17.48%
5cm, 5deg : 44.66%
25cm, 2deg : 89.32%
50cm, 5deg : 94.17%
500cm, 10deg : 96.12%
Evaluate scene KingsCollege: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_Cambridge_KingsCollege.txt
[12/06/2021 21:22:38 pixloc.utils.io INFO] Imported 343 images from pixloc_Cambridge_KingsCollege.txt
[12/06/2021 21:22:38 pixloc.utils.eval INFO]
Median errors: 0.167m, 0.263deg
Percentage of test images localized within:
1cm, 1deg : 0.00%
2cm, 2deg : 0.58%
3cm, 3deg : 1.75%
5cm, 5deg : 7.00%
25cm, 2deg : 66.47%
50cm, 5deg : 82.51%
500cm, 10deg : 97.08%
Evaluate scene GreatCourt: /home/victor/disk1/Disk_E/pixloc/outputs/results/pixloc_Cambridge_GreatCourt.txt
[12/06/2021 22:16:25 pixloc.utils.eval INFO]
Median errors: 0.412m, 0.175deg
Percentage of test images localized within:
1cm, 1deg : 0.00%
2cm, 2deg : 0.00%
3cm, 3deg : 0.92%
5cm, 5deg : 2.50%
25cm, 2deg : 36.05%
50cm, 5deg : 54.08%
500cm, 10deg : 74.21%
from pixloc.
VictorZoo
commented on May 20, 2024
There is another possibility, the checkpoints_best.tar downloaded by python -m pixloc.download --select checkpoints is not the best model. Because in the .log file (megadepth), it displays that there is only 4 epoch. Pleasw cheack into it.
Just a recommendation. For one check way, may you type python -m pixloc.download --select checkpoints and use the downloaded checkpoints_best.tar to see if it matches the results in paper. Just a recommendation.
Sorry to bother you. Thank you!
from pixloc.
sarlinpe
commented on May 20, 2024
7Scenes
It turns out that the default path pointed to the raw SuperPoint+SuperGlue SfM model, while the results reported in the paper are based on the dense depth maps to cleanup the point cloud. This has now been fixed by 0072dc7. Here are the results that you should get:
| Chess | Fire | Heads | Office | Pumpkin | Kitchen | Stairs | |
|---|---|---|---|---|---|---|---|
| Pixloc_author | 2.4/0.81 | 1.9/0.79 | 1.3/0.86 | 2.6/0.79 | 4.1/1.17 | 3.4/1.21 | 4.6/1.22 |
| Paper | 2/0.8 | 2/0.73 | 1/0.82 | 3/0.82 | 4/1.21 | 3/1.2 | 5/1.3 |
Cambridge
Let me run these numbers again.
CMU
Let's track this issue in #20
Aachen
Let's keep this in #23. I feel that this is a similar setup issue as with the CMU dataset.
from pixloc.
sarlinpe
commented on May 20, 2024
These are the results that I obtain for Cambridge Landmarks:
| Court | King’s | Hospital | Shop | St. Mary’s | |
|---|---|---|---|---|---|
| Pixloc code | 28.6/0.15 | 13.0/0.22 | 18.4/0.35 | 4.4/0.22 | 8.7/0.27 |
| Paper | 30/0.12 | 14/0.24 | 16/0.32 | 5/0.23 | 10.0/0.34 |
Again there seems to be a discrepancy between your setup and mine. All experiments were conducted with an RTX 2080 Ti and torch==1.10.0+cu102. Output of pip freeze
from pixloc.
VictorZoo
commented on May 20, 2024
Hi @skydes , thanks for your solution. I changed reference_sfm='{scene}/sfm_superpoint+superglue+depth/' and got the similar results to paper. Detail :
| Chess | Fire | Heads | Office | Pumpkin | Kitchen | Stairs | |
|---|---|---|---|---|---|---|---|
| Pixloc_author | 2.4/0.81 | 1.9/0.78 | 1.3/0.85 | 2.7/0.81 | 4.1/1.16 | 3.4/1.21 | 4.8/1.30 |
| Pixloc_reproduce | 2.4/0.80 | 1.9/0.79 | 1.2/0.87 | 2.7/0.81 | 4.2/1.18 | 3.4/1.20 | 5.2/1.20 |
| Paper | 2/0.8 | 2/0.73 | 1/0.82 | 3/0.82 | 4/1.21 | 3/1.2 | 5/1.3 |
As for Cambridge datasets, I notice that in your last block saids "Pixloc_release". It means the release version not the master version in this repo?
from pixloc.
sarlinpe
commented on May 20, 2024
The evaluation is ran with the master branch of the repository.
from pixloc.
Related Issues (20)
- How to add new query image without reconstruct hloc db? HOT 4
- Dead lock after torch.distributed.all_reduce
- How to generate oracle image retrieval results for Robotcar dataset？
- [Question] Working with black and white images HOT 1
- How to correctly pick 3d points
- Using COLMAP to generate Cambridge landmark scene's poses from scratch HOT 4
- 'CambridgeLandmarks_Colmap_Retriangulated_1024px.zip' this file can't be downloaded HOT 3
- ImportError: cannot import name 'set_logging_debug' HOT 6
- Question: TypeError HOT 4
- Using sfm model on 7scenes without depth HOT 2
- Support for Colmap camera model FULL_OPENCV HOT 2
- With custom data HOT 2
- can not change the encoder of UNet HOT 2
- about the Graphics memory HOT 1
- Some questions about trainning
- some questions about your code in base_optimizer.py
- About colmap's scale proplem HOT 2
- Checkpoint and dataset issue HOT 2
- Can't download 7scenes_sfm_triangulated.zip
- About the inconsistency between 7scenes dataset pose and 7scenes gt sfm pose
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pixloc.