KeyError: 'R_all' about repose HOT 11 CLOSED

Wow, that's awesome! I got it.
Thanks for your detailed explanation.

from repose.

Hi @pyni

Thank you so much for your interest in our work!

I committed a quick fix for the issue, so could you check if that works?
I keep busy with another project and I can not make time to release the training code soon. In fact, it requires some work because I separated the code to extract the PVNet's feature to make evaluation easier. I think you can use the almost same training script as clean-pvnet, so I recommend you to check it out if you want to train the model yourself as soon as possible. I will update this issue again when the training code is available!

Thank you,
Shun

from repose.

Hi @pyni

Thank you so much for your interest in our work!

I committed a quick fix for the issue, so could you check if that works? I keep busy with another project and I can not make time to release the training code soon. In fact, it requires some work because I separated the code to extract the PVNet's feature to make evaluation easier. I think you can use the almost same training script as clean-pvnet, so I recommend you to check it out if you want to train the model yourself as soon as possible. I will update this issue again when the training code is available!

Thank you, Shun

Thanks for replying.
And I have another question about you paper.
I wonder why it should be 'iterative refinement.'?
Since you have learned the deep texture for the 3D model. Why not directly match the CNN Feature with 3D model with learned texture using classical matching algorithms(such as RANSAC+PnP)?
Thanks.

from repose.

You may be confused with the difference between indirect and direct methods. There are generally two ways to optimize a pose --- Minimize 1) the reprojection error (indirect) and 2) the photometric error (direct). For the former one, RANSAC + PnP is used. However, due to the sparse nature of the keypoints, its solution can be inaccurate and vulnerable to the localization error. On the other hand, to minimize photometric (feature-metric) error via non-linear optimization can make use of the information of all the pixels, and be more accurate and robust to the error. Our goal in this paper is object pose "refinement". Therefore, we're using the latter approach. Please refer to this page, BA-Net, and several SLAM related papers for more information.

from repose.

I am sorry that I actually did not answer your question. The minimization problem to match the rendered image of a 3D model and the CNN feature is non-linear and not solvable in a closed-form. That is the reason why iterative non-linear optimization algorithms such as Gauss-Newton and LM need to be used.

from repose.

Hi, sorry to interrupt you again.
I have a doubt about equation (4) in your paper:
e = vec(Finp ) − vec(Frend )

According to the equation (4) in BA-Net(https://arxiv.org/abs/1806.04807):

According to this paper, for each point qj, it should be transformed by dj, Ti and π, then we can extract its feature. But according to equation (4) in your paper, it seems that it has no such transformation and it directly extracts the feature.

So I wonder is it the same?

Thanks.

from repose.

The feature $F_rend$ corresponds to $F_i$. To obtain $F_rend$ a 3D model is transformed by equation 1 and rendered by equation 2 in our paper, so we basically do the almost same thing as equation 4 in BA-Net with equations 1, 2, and 4 in our paper.
If you want to dig into it more, please check out our CUDA code for rendering.

from repose.

OK, I got it. Thanks!

from repose.

No problem!

from repose.

Hi, I want to verify whether the iterative process reduces the objective function loss(equation 5 in your paper), so I add following:

e = f_inp - f_rend
diff_loss = e ** 2
print('objective function loss:',diff_loss.sum(dim=1).sum())

But It seems that the loss is increasing rather than decreasing:

It seems that the iterative object is not minimize equation 5 in your paper but maximize it. So do you know why?

from repose.

Thank you for asking me a question!
Since Jacobian is computed only on the pixels having a projected vertex, the error of the black region of a rendered image should be ignored (we can not calculate the Jacobian in those pixels). You should use r_mask to compute the correct error (e.g., I didn't check the code but e = r_mask * (f_inp - f_rend) should work). We do it inside the CUDA kernel for speeding up, but you don't need to modify it directly. This explanation might be confusing, so let me know if you have any additional questions!

from repose.