Comments (10)
@nikhilaravi yes I mean with 3D loss functions, as in the example you have from a sphere to a dolphin. Yes it would be great adding texture to this example, allowing the object to be rendered, at various points of the deformation, with a sensible texture.
from pytorch3d.
@nikhilaravi Thanks for the update. That would be awesome.
Just so you have an idea of what I am thinking, I assume that I can create some sort of loss that determines if the "scalar values" or in this case a single texture value between close points on mesh 1 matches mesh 2.
pytorch3d seems like a potentially great solution for medical image/mesh analyses (registration to start), but the problem I saw originally with the sphere to dolphin was that its good at making any object appear like another object. So, it's possible that it might move points around and have good looking alignment but one anatomical vertex from mesh 1 might not align with the right anatomical vertex of mesh 2. So, I wondered/thought if assigning values to the vertices that coincide with anatomy could help constrain the registration to ensure that for example the inside of the knee stays as the inside of the knee even after registration.
If this isn't clear but is of use I'm happy to expand more.
The repo is excellent though, and I've already done playing around with methods like those described in the sphere to dolphin as a method of registering meshes together or of just smoothing a mesh surface.
Kudos!
from pytorch3d.
@DemonElite yes the texture is differentiable i.e. if you create a tensor for e.g. vertex_rgb colors which has requires_grad=True
then you should be able to propagate gradients.
We are currently working on adding support for a per-face texture similar to Neural Mesh Renderer.
from pytorch3d.
@nikhilaravi – is there any update on this? Completely understand that it's probably not very high on the priority list right now... just curious as I have a paper waiting in the wings! 😃
from pytorch3d.
@cianeastwood we have added an example tutorial for how to deform and fit a sphere mesh to a target mesh along with texture: https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/fit_textured_mesh.ipynb. Hopefully this helps with your project!
@DemonElite, @gattia, the textures API was refactored in a393296. So now you can store textures as a per face texture map (TexturesAtlas
), or a D dimensional feature per vertex (TexturesVertex
).
from pytorch3d.
@cianeastwood to clarify do you mean source to target mesh deformation with texture using differentiable rendering or with 3D loss functions? The demo in the tutorials is using the 3D mesh losses for the deformation. Do you want to add texture to this example?
from pytorch3d.
is the Texture can be differentiable? like "Example 3: Optimizing textures" in the neural_renderer project
from pytorch3d.
@nikhilaravi
I have a similar question, I think?
I have meshes (typically work on them in vtk) that have scalar values associated with each vertex in the mesh. These values are information about the thickness of cartilage on the mesh (a bone). I am looking to use the thickness (scalar) values in a loss function so that a registration (like the dolphin example) takes these into account.
I cant fully tell from the tex documentation if this is possible. Any thoughts or ideas of where to look would be greatly appreciated.
Thanks!
Anthony.
from pytorch3d.
@gattia we are updating the texturing API such that instead of storing a color per vertex as a texture you can store any D dimensional feature. This might help with your use case. We are currently working on adding support for this.
from pytorch3d.
Please close this issue if you have no further questions!
from pytorch3d.
Related Issues (20)
- can't install in colab HOT 2
- Get intersections between meshes / lines with meshes HOT 1
- Faces list creation does not work on MacOS HOT 2
- Bad rendering results on A100 GPU, but right in V100 or Tesla T4 GPU.
- RuntimeError: The backward pass for this operation requires the 'self' tensor to be strided, but a sparse tensor was given instead. HOT 1
- box3d_overlap may cause “RuntimeError: CUDA error: an illegal memory access was encountered” in some cases HOT 1
- ERROR: Could not build wheels for pytorch3d, which is required to install pyproject.toml-based projects HOT 1
- How to render an image with the correct camera position ?
- Can't install PyTorch3D with Github actions HOT 6
- Rendering point clouds with perspective camera and specified intrinsics with Pulsar backend HOT 2
- explain this erroer message on my miniconda and the best possible code to resolve it ...traceback (most recent call last): File "afy/cam_fomm.py", line 316, in <module> out = predictor.predict(frame) File "C:\Users\mikey\avatarify-python\afy\predictor_local.py", line 102, in predict kp_norm = normalize_kp(kp_source=self.kp_source, kp_driving=kp_driving, File "C:\Users\mikey\avatarify-python\afy\predictor_local.py", line 28, in normalize_kp jacobian_diff = torch.matmul(kp_driving['jacobian'], torch.inverse(kp_driving_initial['jacobian'])) RuntimeError: CUDA error: operation not supported when calling `cusparseCreate(handle)`
- dl.fbaipublicfiles.com dead HOT 1
- Proper MeshRasterizerOpenGL usage HOT 3
- How to add a spot light (cone light) in PyTorch3D for projecting a 2D pattern onto 3D surfaces?
- MeshRenderer gives unexpected result for non-square image_size HOT 1
- Install pytorch3d in aarch64/arm Linux architecture HOT 4
- Add EnvMapLights Class for Environment Map Lighting
- install pytorch3d
- Can't find pytorch3d version 0.7.6 in pytorch3d channel (conda) HOT 1
- RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 4 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch3d.