Inference on custom dataset does not give proper 3d bboxes about egonet HOT 12 CLOSED

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.

A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.

What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

If you mean you want to visualize the 3D boxes in this case, you can draw the 3D boxes with default sizes and the predicted orientation.

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

If you mean you want to visualize the 3D boxes in this case, you can draw the 3D boxes with default sizes and the predicted orientation.

Precisely speaking, what I want to do is to lift the 2D bounding boxes to 3D bounding boxes without 3D proposals. Is this not possible for EgoNet?

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

If you mean you want to visualize the 3D boxes in this case, you can draw the 3D boxes with default sizes and the predicted orientation.

Precisely speaking, what I want to do is to lift the 2D bounding boxes to 3D bounding boxes without 3D proposals. Is this not possible for EgoNet?

If you are not able to prepare 3D proposals. You can still lift 2D predictions to a 3D bounding box by:

1. Run EgoNet and get 2D part coordinates, 3D dimension and orientation predictions.
2. Solve a PnP problem to estimate translation that minimizes re-projection errors. You can refer to the paper "3d bounding box estimation using deep learning and geometry" for details on how this can be done.
   Of course, this would not result in accurate translation estimation compared to an existing 3D proposal.

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

If you mean you want to visualize the 3D boxes in this case, you can draw the 3D boxes with default sizes and the predicted orientation.

Precisely speaking, what I want to do is to lift the 2D bounding boxes to 3D bounding boxes without 3D proposals. Is this not possible for EgoNet?

If you are not able to prepare 3D proposals. You can still lift 2D predictions to a 3D bounding box by:

1. Run EgoNet and get 2D part coordinates, 3D dimension and orientation predictions.
2. Solve a PnP problem to estimate translation that minimizes re-projection errors. You can refer to the paper "3d bounding box estimation using deep learning and geometry" for details on how this can be done.
   Of course, this would not result in accurate translation estimation compared to an existing 3D proposal.

So, you are suggesting that EgoNet alone cannot estimate 3D bounding boxes from 2D predictions?

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

If you mean you want to visualize the 3D boxes in this case, you can draw the 3D boxes with default sizes and the predicted orientation.

Precisely speaking, what I want to do is to lift the 2D bounding boxes to 3D bounding boxes without 3D proposals. Is this not possible for EgoNet?

If you are not able to prepare 3D proposals. You can still lift 2D predictions to a 3D bounding box by:

1. Run EgoNet and get 2D part coordinates, 3D dimension and orientation predictions.
2. Solve a PnP problem to estimate translation that minimizes re-projection errors. You can refer to the paper "3d bounding box estimation using deep learning and geometry" for details on how this can be done.
   Of course, this would not result in accurate translation estimation compared to an existing 3D proposal.

So, you are suggesting that EgoNet alone cannot estimate 3D bounding boxes from 2D predictions?

My answer is EgoNet can estimate 3D bounding boxes from 2D predictions. The above two steps already describe what to do in this case. It seems you are not capable of understanding the two steps I mentioned. Step two is contained in some legacy code which you may refer to:
#9 (comment)

This is not an issue and you can ask further questions in the discussion part of this repo.

from egonet.

Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.
A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95

However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.
What should I do?

Hi, EgoNet only updates orientation and does not modify or add 3D location/dimension information. Using 2D boxes as proposals by default only updates orientations. You can use 3D box proposals as inputs instead.

Then, what should I do if I want to generate 3D bounding boxes from 2D bounding boxes and orientation information? Are any options to config files to do so?

If you mean you want to visualize the 3D boxes in this case, you can draw the 3D boxes with default sizes and the predicted orientation.

Precisely speaking, what I want to do is to lift the 2D bounding boxes to 3D bounding boxes without 3D proposals. Is this not possible for EgoNet?

If you are not able to prepare 3D proposals. You can still lift 2D predictions to a 3D bounding box by:

1. Run EgoNet and get 2D part coordinates, 3D dimension and orientation predictions.
2. Solve a PnP problem to estimate translation that minimizes re-projection errors. You can refer to the paper "3d bounding box estimation using deep learning and geometry" for details on how this can be done.
   Of course, this would not result in accurate translation estimation compared to an existing 3D proposal.

So, you are suggesting that EgoNet alone cannot estimate 3D bounding boxes from 2D predictions?

My answer is EgoNet can estimate 3D bounding boxes from 2D predictions. The above two steps already describe what to do in this case. It seems you are not capable of understanding the two steps I mentioned. Step two is contained in some legacy code which you may refer to: #9 (comment)

This is not an issue and you can ask further questions in the discussion part of this repo.

Alright. Thanks. Sorry if I dragged this issue too long.

from egonet.

This may be a stupid question, but can we feed in results of the pnp algorithm back into Egonet itself to obtain a refined 3D box prediction?
So if our input is a 2D image and bounding box, Egonet gives us the orientation and dimensions only as it is 2D. We then solve the PnP algorithm to get a 3D bounding box estimate, and then feed that back into Egonet to refine this prediction.

from egonet.

This may be a stupid question, but can we feed in results of the pnp algorithm back into Egonet itself to obtain a refined 3D box prediction? So if our input is a 2D image and bounding box, Egonet gives us the orientation and dimensions only as it is 2D. We then solve the PnP algorithm to get a 3D bounding box estimate, and then feed that back into Egonet to refine this prediction.

Yes. You can treat the results of the pnp algorithm as another input 3D box proposal, compute a 2D RoI from it and feed that to Ego-Net. The orientation predictions may fluctuate a bit but not too much if the 2D RoI inputs are similar.

from egonet.

Thanks. So just to confirm that after the pnp algorithm we do not use original 2D ROI, but compute this from the 3D ROI produced by pnp?

from egonet.

Thanks. So just to confirm that after the pnp algorithm we do not use original 2D ROI, but compute this from the 3D ROI produced by pnp?

Yes, During training the 2D RoIs are computed from ground truth 3D boxes.

from egonet.