mworchel / svbrdf-estimation Goto Github PK

The crash occurs on a device sync after a thrust operation, but could be caused invalid operations prior to the thrust kernel. We have to investigate that.

Backpropagation on CPU does not work as expected. When running the pose_estimation in Google Colab, the loss does not really converge and under Windows the loss varies randomly. See BachiLi/redner#93 for reference.

Since the input images are artificially generated for each training sample, we should find out if they are they same for each sample and each epoch (essentially the data loader repeats the data set) or if they are different (data loader queries the dataset again).

When generating artificial input images, the reference code adds gaussian noise to the renderings to simulate noise of the camera sensor. We should support this as well, although it's low priority.

See file helpers.py line 256 for reference.

During training, the model should be saved all x epochs in order to have a valid state if an exception occurs and not losing all progress.

Add support for material mixing. It combines the SVBRDF maps of multiple different materials into a set of new maps, creating a mixture of both materials.

See dataReader.py line 141 for reference.

While the weights of the network and the current epoch are saved correctly, we do not store the optimizer (and scheduler) state right now. In order to continue training predictably, we should do so. This can serve as a reference https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training. It also shows, how extra metadata like the epoch and current loss can be stored in the checkpoint file.

Add a model (checkpoint) loading capability to the main script. This requires the possibility to load the model itself but also metadata about the training, like current epoch etc.

We should add a parameter like model-dir or something.

Hi,
thanks for this cool PyTorch-port! Do you have a trained checkpoint for the SingleView Model available? That'd be great!

Thanks, and best

Dear Author,
I like your interesting work and I find the GitHub explanation very nice. I wanted to try the code but I fail to create the environment. There is no support for the Redner-GPU package and I tried every way to use it. Pip, conda, mamba all of them. Here is the error attached. Do you have any solution? I would really appreciate your help!

Thank you,

Currently, only a full retraining of the model is possible. We should implement means to continue training that has been interrupted before. To do this, we need to store metadata along the model, which contains information about the current state of the training (e.g., number of finished epochs).

The rendering loss currently uses a fixed set of three scenes. We should use semi randomized scenes as in the reference implementation, where several diffuse and several specular configurations are generated.

Thank you very much for providing open source code, but why is there color difference in the picture rerendered by the rendering program? How can I solve it?

Currently, we only support reading samples with maps and images of size 256x256. We have to support arbitrarily sized inputs in order to read the training data and crop them down to 256x256 to be able to use them with the model.

We implemented the coordinate layers as given in the implementation but never tested the network with it. We should definitely do so.

Support spot lights in the LocalRenderer, i.e., use a specific flash falloff instead of the usual inverse quadratic falloff. This is particularly interesting for input image generation and could result in better artificial training data.

The keyword in the reference code is currentConeTargetPos and relevant lines are line 85-89 and 124-131 in renderer.py.

We have the code to perform explicit weight initialization but it's currently not used. The reference uses it so it would be interesting which impact it has also on convergence.

Currently, the loss is submitted to Tensorboard with the epoch as index. Since we can have multiple batches per epoch, the loss is effectively only reported once per epoch (at the end), as all previous values are overwritten. We should report it with an increasing index based on the current epoch and dataset size.

The reference code uses jittering of light and view positions, when generating artificial input images. We should support this as well, although it's low priority.

See file helpers.py line 236 for reference.

Add support for the scene movingViewHemiSpotLightOneSurface as in the reference code

Training and testing should be individual modes of the executable so they can be executed without one another. This is required for making the training headless, i.e., enable it to be executed on a remote machine without display.

Hello Markus! Thank you for sharing this great project. To get started quickly, is it possible you could share pre-trained weights as well? I am particularly interested in the MultiViewModel.

Thank you!