fluxml / flux3d.jl Goto Github PK

Once #23 is merged, for better and continued maintainance we should be moving the repo to FluxML.

cc @DhairyaLGandhi

Ref https://arxiv.org/pdf/2002.10099.pdf

It would be good to have a working example of this paper in the repo and perhaps see what is needed to get it to train. Getting PDEs to train with NeuralPDE might be the way to go here.

cc @avik-pal

Hi I'm new to Julia, so this is possibly just my mistake, but I cant seem to load the datasets from the tutorials:

If I execute (from https://fluxml.ai/Flux3D.jl/dev/datasets/modelnet/)
dset = ModelNet10(train=false)
I get the error:
dataset not found and auto-download option is set false.

And if I run:

dset = ModelNet10.dataset(;
    mode = :pointcloud,
    npoints = npoints,
    transform = NormalizePointCloud(),
)

I get the error:
type #ModelNet10 has no field dataset

I am planning to rework RayTracer.jl to make use of Flux3D for its handling of meshes. I am wondering how difficult would it be to handle textures.

There are two components to this:

• Vertex Colors
• Image Textures

I believe the first one is just storing a 4D Array. The other one is slightly more complicated to handle though.

I'm interested in a fast GPU implementation for k nearest neighbor queries -- similar to pytorch3d.ops.knn_points. Flux3D seems like an appropriate place for this to live.

Relevant pytorch3d code:
pytorch3d/csrc/knn
pytorch3d/csrc/utils/mink.cuh

Relevant Julia code:
https://github.com/JuliaParallel/rodinia/blob/master/julia_cuda/nn/nn.jl

1. We don't use separate modules for datasets any longer, so would be great to update those.
2. We need documentation for the Voxel representation

Once these are in we should tag a new release along with the blog post.

The README currently lacks any proper detail about the package. I would recommend adding the following at the earliest

• Installation Instructions
• Tiny code snippet for showing off the package
• A cool visual demo (the dolphin demo is pretty interesting so that should be good)
• Motivation behind the package
• Ordered list of examples / tutorials

We need to perform performance regressions on the mesh representation. The first step here would be to benchmark against Pytorch3D and Kaolin.

PointFlow is a really interesting application of both Point Clouds and CNFs, and will as a great demo from a "marketing" perspective.

The current blocker for this is on the DiffEqFlux.jl side SciML/DiffEqFlux.jl#342. Reposting what @nirmal-suthar pointed out on Julia Slack:

For CNF layer in PointFlow model, I am using FFJORD from DiffEqFlux. Currently, I am having some trouble with matching generated data with original distribution after training dummy CNF layer. I have discussed this issue with the one who wrote this layer and will get back to this issue in some time. Additionally, this layer also lacks batched format, which is also a serious problem as training a single pointcloud of 1000 points for 10 epochs takes ~10 hours. I tried fixing this for a forward pass, but zygote gave some weird error.

Loading the ModelNet10 data, I found one of the data (I=2431) caused the error if I convert it to PointCloud, although the mesh can be visualized. The error is at Categorical: the condition isprobvec(p) is not satisfied.

julia> using Flux3D

julia> dset = ModelNet10(;
root = "/Users/kahingleung/Downloads/ModelNet10",
train = true,
download = false,

   )

ModelNet Dataset:
root: ModelNet10
train: true
length: 3991
transform: nothing
categories: 10

julia> m = dset[2431].data
TriMesh{Float32, UInt32, Array} Structure:
Batch size: 1
Max verts: 4182
Max faces: 3731
offset: -1
Storage type: Array

julia> using Makie

julia> visualize(m)
[ Info: Makie/AbstractPlotting is caching fonts, this may take a while. Needed only on first run!

julia> p = PointCloud(m)
ERROR: ArgumentError: Categorical: the condition isprobvec(p) is not satisfied.
Stacktrace:
[1] macro expansion at /Users/kahingleung/.julia/packages/Distributions/jFoHB/src/utils.jl:6 [inlined]
[2] #_#37 at /Users/kahingleung/.julia/packages/Distributions/jFoHB/src/univariate/discrete/categorical.jl:30 [inlined]
[3] #Categorical#38 at /Users/kahingleung/.julia/packages/Distributions/jFoHB/src/univariate/discrete/categorical.jl:34 [inlined]
[4] Distributions.DiscreteNonParametric{Int64,P,Base.OneTo{Int64},Ps} where Ps where P(::Array{Float64,1}) at /Users/kahingleung/.julia/packages/Distributions/jFoHB/src/univariate/discrete/categorical.jl:34
[5] sample_points(::TriMesh{Float32,UInt32,Array}, ::Int64; eps::Float32) at /Users/kahingleung/.julia/packages/Flux3D/Kv1eP/src/transforms/mesh_func.jl:45
[6] sample_points at /Users/kahingleung/.julia/packages/Flux3D/Kv1eP/src/transforms/mesh_func.jl:26 [inlined]
[7] PointCloud at /Users/kahingleung/.julia/packages/Flux3D/Kv1eP/src/conversions.jl:41 [inlined] (repeats 2 times)
[8] top-level scope at REPL[7]:1

It seems AbstractPlotting.vbox (which is used in this docs ) is removed due to this commit MakieOrg/Makie.jl@fac052d we need to find alternatives that achieve this functionality.

I've implemented visualize! function that is similar to Flux3D.visualize except it accepts axis3::Axis3.

See gist https://gist.github.com/terasakisatoshi/00225d1dd053e1516e36ef97a9115ef5 to see the complete code.

If you like it, I will send a PR.

This is particularly useful in rendering applications where you need different projections for meshes which don't follow the same alignment

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

The current documentation does a good job of listing the API but is difficult for a beginner to parse. Listing down a few improvements here. Will update the list as we go forward:

1. Add more structure to the docs.
   • There must be clear instructions to install the package.
   • Separate pages for different representations
   • Explain the Mesh representation a bit. It is not clear how the different formats are useful.
   • Have a different page listing the available set of metrics/loss functions
2. The examples should be displayed in the Documentation itself, and follow a format where the entire executable code is listed at top and followed by a step-by-step explaination. Look at https://diffeqflux.sciml.ai/dev/ for inspiration.

I tied copy-pasting the example code and got ERROR: UndefVarError: visualize not defined
I also tied julia ? visualize and got
Couldn't find visualize
Perhaps you meant visualize, finalize or isvalid
No documentation found.

Binding visualize does not exist.

Copying the function from src works though.

chamfer_loss uses verts_padded representation for computing distance, need workaround to make verts_packed representation of trimesh visible to Zygote.

EDIT: use custom adjoint for computing gradient for verts_packed