mworchel / svbrdf-estimation Goto Github PK
View Code? Open in Web Editor NEWSVBRDF Estimation using a Physically-based Differentiable Renderer
License: MIT License
SVBRDF Estimation using a Physically-based Differentiable Renderer
License: MIT License
The crash occurs on a device sync after a thrust operation, but could be caused invalid operations prior to the thrust kernel. We have to investigate that.
Backpropagation on CPU does not work as expected. When running the pose_estimation
in Google Colab, the loss does not really converge and under Windows the loss varies randomly. See BachiLi/redner#93 for reference.
Since the input images are artificially generated for each training sample, we should find out if they are they same for each sample and each epoch (essentially the data loader repeats the data set) or if they are different (data loader queries the dataset again).
When generating artificial input images, the reference code adds gaussian noise to the renderings to simulate noise of the camera sensor. We should support this as well, although it's low priority.
See file helpers.py
line 256 for reference.
During training, the model should be saved all x
epochs in order to have a valid state if an exception occurs and not losing all progress.
Add support for material mixing. It combines the SVBRDF maps of multiple different materials into a set of new maps, creating a mixture of both materials.
See dataReader.py
line 141 for reference.
While the weights of the network and the current epoch are saved correctly, we do not store the optimizer (and scheduler) state right now. In order to continue training predictably, we should do so. This can serve as a reference https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training. It also shows, how extra metadata like the epoch and current loss can be stored in the checkpoint file.
Add a model (checkpoint) loading capability to the main script. This requires the possibility to load the model itself but also metadata about the training, like current epoch etc.
We should add a parameter like model-dir
or something.
Hi,
thanks for this cool PyTorch-port! Do you have a trained checkpoint for the SingleView Model available? That'd be great!
Thanks, and best
Dear Author,
I like your interesting work and I find the GitHub explanation very nice. I wanted to try the code but I fail to create the environment. There is no support for the Redner-GPU package and I tried every way to use it. Pip, conda, mamba all of them. Here is the error attached. Do you have any solution? I would really appreciate your help!
Currently, only a full retraining of the model is possible. We should implement means to continue training that has been interrupted before. To do this, we need to store metadata along the model, which contains information about the current state of the training (e.g., number of finished epochs).
The rendering loss currently uses a fixed set of three scenes. We should use semi randomized scenes as in the reference implementation, where several diffuse and several specular configurations are generated.
Thank you very much for providing open source code, but why is there color difference in the picture rerendered by the rendering program? How can I solve it?
Currently, we only support reading samples with maps and images of size 256x256. We have to support arbitrarily sized inputs in order to read the training data and crop them down to 256x256 to be able to use them with the model.
We implemented the coordinate layers as given in the implementation but never tested the network with it. We should definitely do so.
Support spot lights in the LocalRenderer
, i.e., use a specific flash falloff instead of the usual inverse quadratic falloff. This is particularly interesting for input image generation and could result in better artificial training data.
The keyword in the reference code is currentConeTargetPos
and relevant lines are line 85-89 and 124-131 in renderer.py
.
We have the code to perform explicit weight initialization but it's currently not used. The reference uses it so it would be interesting which impact it has also on convergence.
Currently, the loss is submitted to Tensorboard with the epoch as index. Since we can have multiple batches per epoch, the loss is effectively only reported once per epoch (at the end), as all previous values are overwritten. We should report it with an increasing index based on the current epoch and dataset size.
The reference code uses jittering of light and view positions, when generating artificial input images. We should support this as well, although it's low priority.
See file helpers.py
line 236 for reference.
Add support for the scene movingViewHemiSpotLightOneSurface
as in the reference code
Training and testing should be individual modes of the executable so they can be executed without one another. This is required for making the training headless, i.e., enable it to be executed on a remote machine without display.
Hello Markus! Thank you for sharing this great project. To get started quickly, is it possible you could share pre-trained weights as well? I am particularly interested in the MultiViewModel
.
Thank you!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.