Comments (5)
Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?
Hi, to use EgoNet on your custom data:
- You need to prepare 2D or 3D proposals for your images before using EgoNet.
- Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.
from egonet.
Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?
Hi, to use EgoNet on your custom data:
- You need to prepare 2D or 3D proposals for your images before using EgoNet.
- Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.
Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.
from egonet.
Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?
Hi, to use EgoNet on your custom data:
- You need to prepare 2D or 3D proposals for your images before using EgoNet.
- Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.
Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.
You can use the predicted cuboid ("kpts_3d_pred") to visualize the orientation arrow instead. In this way, you don't need a 3D box input.
from egonet.
Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?
Hi, to use EgoNet on your custom data:
- You need to prepare 2D or 3D proposals for your images before using EgoNet.
- Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.
Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.
You can use the predicted cuboid ("kpts_3d_pred") to visualize the orientation arrow instead. In this way, you don't need a 3D box input.
But the below code for adding an orientation arrow can be possible only by using both 'kpts_3d_gt' and 'kpts_3d_pred'. I understand that Keypoints are different from 3D bounding boxes, but if I only use 'resources/text_boxes' prediction files, that just have the correct 2D bounding box location then the 'annot_dict' contains 'kpts_3d_before' and not 'kpts_3d_gt'. Thus, not able to get the orientation arrow. So I will have to use the txt files that also have the correct 6 fields (other than 2D bboxes) related to the 3D coordinate system for the code to have 'kpts_3d_gt'. As when I use the output files from D4LCN, the 'annot_dict' has 'kpts_3d_gt', which gives me an orientation arrow (Adding the difference in image visualizations below).
Please let me know if I understood this correctly or not. Thank you for the help and sorry for so many questions!
def add_orientation_arrow(self, record):
"""
Generate an arrow for each predicted orientation for visualization.
"""
pred_kpts = record['kpts_3d_pred']
gt_kpts = record['kpts_3d_gt']
K = record['K']
arrow_2d = np.zeros((len(pred_kpts), 2, 2))
for idx in range(len(pred_kpts)):
vector_3d = (pred_kpts[idx][1] - pred_kpts[idx][5])
arrow_3d = np.concatenate([gt_kpts[idx][0].reshape(3, 1),
(gt_kpts[idx][0] + vector_3d).reshape(3, 1)],
axis=1)
projected = K @ arrow_3d
arrow_2d[idx][0] = projected[0, :] / projected[2, :]
arrow_2d[idx][1] = projected[1, :] / projected[2, :]
# fix the arrow length if not fore-shortened
vector_2d = arrow_2d[idx][:,1] - arrow_2d[idx][:,0]
length = np.linalg.norm(vector_2d)
if length > 50:
vector_2d = vector_2d/length * 60
arrow_2d[idx][:,1] = arrow_2d[idx][:,0] + vector_2d
return arrow_2d
from egonet.
Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?
Hi, to use EgoNet on your custom data:
- You need to prepare 2D or 3D proposals for your images before using EgoNet.
- Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.
Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.
You can use the predicted cuboid ("kpts_3d_pred") to visualize the orientation arrow instead. In this way, you don't need a 3D box input.
But the below code for adding an orientation arrow can be possible only by using both 'kpts_3d_gt' and 'kpts_3d_pred'. I understand that Keypoints are different from 3D bounding boxes, but if I only use 'resources/text_boxes' prediction files, that just have the correct 2D bounding box location then the 'annot_dict' contains 'kpts_3d_before' and not 'kpts_3d_gt'. Thus, not able to get the orientation arrow. So I will have to use the txt files that also have the correct 6 fields (other than 2D bboxes) related to the 3D coordinate system for the code to have 'kpts_3d_gt'. As when I use the output files from D4LCN, the 'annot_dict' has 'kpts_3d_gt', which gives me an orientation arrow (Adding the difference in image visualizations below). Please let me know if I understood this correctly or not. Thank you for the help and sorry for so many questions!
def add_orientation_arrow(self, record): """ Generate an arrow for each predicted orientation for visualization. """ pred_kpts = record['kpts_3d_pred'] gt_kpts = record['kpts_3d_gt'] K = record['K'] arrow_2d = np.zeros((len(pred_kpts), 2, 2)) for idx in range(len(pred_kpts)): vector_3d = (pred_kpts[idx][1] - pred_kpts[idx][5]) arrow_3d = np.concatenate([gt_kpts[idx][0].reshape(3, 1), (gt_kpts[idx][0] + vector_3d).reshape(3, 1)], axis=1) projected = K @ arrow_3d arrow_2d[idx][0] = projected[0, :] / projected[2, :] arrow_2d[idx][1] = projected[1, :] / projected[2, :] # fix the arrow length if not fore-shortened vector_2d = arrow_2d[idx][:,1] - arrow_2d[idx][:,0] length = np.linalg.norm(vector_2d) if length > 50: vector_2d = vector_2d/length * 60 arrow_2d[idx][:,1] = arrow_2d[idx][:,0] + vector_2d return arrow_2d
Hi, the gt_kpts were used only to get the translation so that the arrow can be computed in the camera coordinate system.
- If you have a 3D proposal, you can simply modify the code to use the proposed translation as well.
- If you only have a 2D proposal, you can draw the arrow in the Bird's Eye View. Otherwise, you can approximate the 2D arrow with the predicted orientation.
from egonet.
Related Issues (17)
- Running the model on a single image HOT 5
- About the license for this model HOT 2
- Visualazition problem
- Generate 3D rectangular coordinates using 2D rectangular boxes HOT 4
- Inference on custom dataset does not give proper 3d bboxes HOT 12
- Relation between kpts_3d_pred and pose HOT 2
- Result Image HOT 1
- 关于“car_instance”中代码的请教 HOT 1
- 如何使用该模型推断自己的图片
- 如何使用该模型推断自己的图片 HOT 4
- Cannot use my predicted files to run the inference.py
- Training Script HOT 2
- Reproduce Result in Kitti dataset HOT 1
- Reproduce results on the test split HOT 2
- No arg_max in KITTI_train_ICRs.yml file HOT 1
- Stream from camera HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from egonet.