Rendering Instance Masks about pyrender HOT 10 OPEN

Here's an efficient way to do it all in the same renderer!

# Assume we have an offscreen renderer and a scene
segimg = np.zeros((height, width), dtype=np.uint8)
flags = RenderFlags.DEPTH_ONLY

# Render full depth
full_depth = renderer.render(scene, flags=flags)

# Hide all mesh nodes
for mn in scene.mesh_nodes:
    mn.mesh.is_visible = False

# Now, iterate through them, enabling each one
for i, node in enumerate(scene.mesh_nodes):
    node.mesh.is_visible = True
    depth = renderer.render(scene, flags=flags)
    mask = np.logical_and(
        (np.abs(depth - full_depth) < 1e-6), np.abs(full_depth) > 0
    )
    segimg[mask] = i + 1
    node.mesh.is_visible = False

# Show all meshes again
for mn in scene.mesh_nodes:
    mn.mesh.is_visible = True  

This bit of code will produce a segimg containing what you want. Does that work for you, or do you still want flat shading?

from pyrender.

HI @mmatl! Thanks a lot for this cool library :)
Did you have time already to look into this issue again?

from pyrender.

Just found out that the instance segmentation is supported since version 0.1.41.

You can also find an example in the example.py

TL;DR:

from pyrender import RenderFlags
nm = {node: 20*(i + 1) for i, node in enumerate(scene.mesh_nodes)}   # Node->Seg Id map
seg = r.render(scene, RenderFlags.SEG, nm)[0]

from pyrender.

Sorry y'all! Got sidetracked at work. Will try to push this out soon. It should be pretty easy to write a shader to do exactly this in a single forward pass.

from pyrender.

Hi! The standard way to render instance masks actually is to render individual depth images and do a depth difference between each depth image and the overall image, as you've been doing so far. In principle, it's possible to render instance masks in a single pass if you do totally flat shading and give each mesh a unique color, but for now, pyrender doesn't support totally flat shading. I can add it in as an option if you think it'd be useful, though. I haven't been having performance issues with the depth differencing method myself :)

from pyrender.

@mmatl Thanks a lot! This is helpful but may not be enough.

I'm trying to render instance masks for video sequences from a big 3D dataset containing 1500 scenes. This roughly amounts to a total of 2.5 million images.

If I have to render each frame 20-30 times (once for each object/mesh), it seems (based on a few experimental runs) the whole thing might take up to a month. With flat-shading, however, it would be 1-2 days which is more reasonable.

It would be good to have a flat shading option. I'd be happy to help you with the implementation with some guidance!

from pyrender.

@shubham-goel Hmm after thinking about this for a bit, I think that there may be a better way to do this. It's probably better to write a custom vertex + fragment shader and just give you the option to use those in the rendering flags. Let me try to mock that up real fast.

Going to be out and about during the day today, but I should be able to get something to you tonight or tomorrow.

from pyrender.

Also, do run multiple processes at once! Your GPU/CPU should be able to handle it, and this sort of task is definitely easily parallelizable.

from pyrender.

@shubham-goel Hmm after thinking about this for a bit, I think that there may be a better way to do this. It's probably better to write a custom vertex + fragment shader and just give you the option to use those in the rendering flags. Let me try to mock that up real fast.

Going to be out and about during the day today, but I should be able to get something to you tonight or tomorrow.

Thanks for the awesome library. Did you add this feature? If so, could you please share an example for that?

from pyrender.

Just found out that the instance segmentation is supported since version 0.1.41.

You can also find an example in the example.py

TL;DR:

from pyrender import RenderFlags
nm = {node: 20*(i + 1) for i, node in enumerate(scene.mesh_nodes)}   # Node->Seg Id map
seg = r.render(scene, RenderFlags.SEG, nm)[0]

This did the job for me, thanks for sharing!

from pyrender.