Can this work with purely synthetic training data (and no texture map)? about repose HOT 5 CLOSED

Since RePOSE heavily relies on a visual feature from an observed image, I honestly think it's not that easy to make it work.
I like your idea and the settings of no images with real texture are realistic, so it should be an interesting future research direction!

from repose.

Hi @murrdpirate! Thank you so much for your interest in our project!

In some cases, there are also no real images of the object to train on. In these cases, synthetic data with domain randomization is often used to render images with random textures applied to the object's 3D model.

The important thing is that creating synthetic images still requires a textured 3D model. RePOSE can perform decently well even trained on only real images (no texture map is needed at all) but we needed to use some synthetic images for a fair comparison with other methods.

If I understand correctly, RePOSE trains on real images of the object. Have you done any experimentation with just domain-randomized synthetic data? Any intuition on whether this will work?

As mentioned above, it is not ideal but we use both synthetic and real images. I think RePOSE should work if you synthesize images with a high-quality textured 3D model, but as you said it will lead to another challenge for RGB scanning. We actually propose an efficient way to collect large-scale real images for 3D object detection in another paper. It would be helpful if you want to tackle several challenging object categories hard to obtain an RGB texture.

from repose.

Thanks for the reply @sh8!

The important thing is that creating synthesis images still requires a textured 3D model.

What I'm wondering is: what if I did not have a texture for the model, and so I just applied random textures when rendering synthetic images? Would that have any hope in working with this procedure?

It's definitely a harder problem when no images with real texture are available, but I think it's still solvable.

from repose.

Appreciate the feedback!

from repose.

No problem! Please ask me any questions anytime!

from repose.