Started work with an off-screen PyOpenGL + GLUT (invisible) window which can be rendered to. The idea is to use that canvas to draw the OpenGL related items to before the model forward render. Then, the OpenGL rendered data (RGB + A + Depth) are all passed into the GS rendering backend with compositing working properly given the other OpenGL render. Splats in front of the depth buffer will splat as usual, but after the depth buffer will not be splatted. Instead, the RGBA from the OpenGL render will be composited, and splatting will only continue if the alpha is not saturated yet. Should work well with only a little overhead on backend.

So far I can render a cube outline using PyOpenGL to a hidden window and also obtain the depth buffer (normalized 0.0-1.0 for near/far plane), and obtain both these arrays as numpy/standard float arrays for passing to the GS CUDA backend.

Next steps

1. Implement CUDA rasterizer updates to properly draw gaussians on top of/behind the given RGBAD buffers given to the rasterizer.
2. Update OpenGL renderer to maintain a consistent view/projection matrix to the GS renderer model such that the buffer objects are pixel-wise accurate with each other per frame.
3. Build out the OpenGL renderer with support for arbitrary objects (wireframe rectangles for cameras, images as textures for debug viewing, etc) so the main code can call high-level functions easily.

Useful Links

• https://stackoverflow.com/questions/52036176/pyopengl-get-depth-map-of-drawn-image
• https://pyopengl.sourceforge.net/context/tutorials/nehe2.html

from gstk.

Updates:

• Creates a frameBufferObject (FBO) to render to, which ignores the frame buffer for the actual window output. Truly "off-screen" rendering. Able to read the colors and depths still
• Uses glLoadMatrixf to load the exact modelview matrix and projection matrix, so the OpenGL camera has the exact same parameters as the gaussian splatting one. Had to reverse the z-sign for the projection matrix, seems GS and OpenGL use opposite z-signs (camera forward)
• Reorganized the OpenGL_renderer into an object that can be rendered with the render_cam object we use to render the GS model. Should be trivial to add opengl_renderer.render(render_cam) in the main render thread.

from gstk.

Updates:

• Moves OpenGL context to glfw. More modern and nicer than GLUT it seems. Plus, GLFW actually allows a fully hidden window for truly off-screen rendering.
• Properly renders depth + rgba in opengl for use later
• Adjusted CUDA code for GS to accept an RGBA + depth buffer. While traversing splats (per pixel), will properly composite OpenGL rendered objects behind/in front of splats as needed.
• OpenGL camera is now the exact same as the GS model at all times.

Left TODO:

• Still need to build out backend OpenGL objects for visualization. At the moment, only a wireframe cube is available. Will begin with this for the selector editor

from gstk.