How to train the model on a single class of Shapenet Data? about differentiable_volumetric_rendering HOT 14 CLOSED

Hi @nitish11 , thanks a lot for your interest in our project!

Single Class Training:
You can train on a single class of the ShapNet dataset by adjusting the data - classes argument in your config. For example, if you only want to train on cars which is the class "02958343", you have to set

data:
  classes: ['02958343']

What is the Pointcloud Chamfer file?
For evaluating our and baseline methods, we report the Chamfer distance. The pointcloud chamfer file entry in the config defines the name of the file containing the ground truth points which can be found in the respective model folder. We load the respective field with this name as an argument for evaluation.

from differentiable_volumetric_rendering.

Hi @nitish11,

yes, you are getting the error because you are not using a correct config file. The NMR Dataset you downloaded is only used by the RGB model which can be trained using this config file.

img_folder_input is the name of the image folder; In the NMR Dataset, it is "image".

If you want to train the RGB model with on the category "03691459" from the NMR Dataset, you can add

  classes: ['02958343']

under data of this config file, but you have to keep the other options as they are.

To be completely clear, you end up with a config file with the following config:

data:
  dataset_name: NMR
  path: data/NMR_Dataset
  train_split: softras_train
  val_split: softras_val
  test_split: softras_test
  img_folder_input: image
  img_extension_input: png
  depth_from_visual_hull: true
  depth_range: [0., 5.]
  classes: ['02958343']
model:
  decoder_kwargs:
    hidden_size: 128
training:
  out_dir:  out/single_view_reconstruction/multi_view_supervision/ours_rgb
  n_workers: 8
  model_selection_metric: mask_intersection
  model_selection_mode: maximize
  n_eval_points: 4096

Please check this section of our readme for how to start training a model. I hope this helps!

from differentiable_volumetric_rendering.

Hi @m-niemeyer, thanks fro your reply and clarifying the doubt.

I want to train a model on only one class,; i.e; speaker - 03691459 . Before downloading the data using your script, I just wanted to confirm if Kato shapenet data contains Speaker class or not. Could you please confirm.

from differentiable_volumetric_rendering.

@nitish11 yes, both ShapeNet datasets (Kato renderings and Choy renderings) contain the speaker class.

from differentiable_volumetric_rendering.

@m-niemeyer : Thanks for the confirmation, I downloaded the NMR_dataset. And copied only one folder 03691459. I modified config/default.yaml as mentioned below.

method: dvr
data:
  dataset_name: Shapes3D
  path: data/ShapeNet
  classes: ['03691459']
  input_type: image
  dim: 3
  n_views: 0
  n_views_input: 0
  pointcloud_chamfer_file: pointcloud.npz
  depth_folder: depth
  img_folder: image
  visual_hull_depth_folder: visual_hull_depth
  mask_folder: mask
  img_extension: png
  img_extension_input: jpg
  depth_extension: exr
  mask_extension: png
  img_size: null

What is the use of "img_folder_input"?
https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/default.yaml#L13 ?

I am getting the error Error occurred when loading field inputs of model c3233311a94b7b968a0181490cb73cbc (03691459), the variables that are fetched at this point are as follows

extension: jpg
folder: data/ShapeNet/03691459/c3233311a94b7b968a0181490cb73cbc/img_choy2016
files: []

The dataset contains

$ ls data/ShapeNet/03691459/c3233311a94b7b968a0181490cb73cbc
image mask visual_hull_depth cameras.npz pointcloud.npz

from differentiable_volumetric_rendering.

@m-niemeyer : Thanks for the explanation.

Now, I am getting error in reading depth_from_visual_hull.

For filename : data/NMR_Dataset/03691459/dc9c9ce6dc9a771f500158c23c4c5a8e/visual_hull_depth/00060001.exr

I am getting below error.

in load_visual_hull_depth
    depth = np.array(imageio.imread(filename)).astype(np.float32)
  File "/home/nitish/anaconda3/envs/s3d/lib/python3.8/site-packages/imageio/core/functions.py", line 265, in imread
    reader = read(uri, format, "i", **kwargs)
  File "/home/nitish/anaconda3/envs/s3d/lib/python3.8/site-packages/imageio/core/functions.py", line 181, in get_reader
    raise ValueError(
ValueError: Could not find a format to read the specified file in single-image mode

from differentiable_volumetric_rendering.

I think this might be related to your imageio library which cannot read exr files. Can you please have a look at this issue?

In particular, I think adding

import imageio
imageio.plugins.freeimage.download()

to e.g. train.py should fix the problem!

from differentiable_volumetric_rendering.

Thanks for the suggestion.

import imageio
imageio.plugins.freeimage.download()

But we need to update the above lines here in im2mesh/data/fields.py.

from differentiable_volumetric_rendering.

Hi @nitish11 @m-niemeyer ,
Thanks for your clear and elaborative discussion that helped me a lot.

I could be able to train the model for a custom single class using this config file -
method: dvr
data:
dataset_name: NMR
path: /home/ujjawal/my_work/object_recon/differentiable_volumetric_rendering/data/j_data
classes: ['00']
input_type: image
dim: 3
n_views: 0
n_views_input: 0
pointcloud_chamfer_file: pointcloud.npz
depth_folder: depth
img_folder: images
img_folder_input: images
visual_hull_depth_folder: depth
mask_folder: mask
img_extension: png
img_extension_input: png
depth_extension: png
mask_extension: png
img_size: null
img_size_input: 224
img_with_camera: true
img_with_mask: true
train_split: train
val_split: val
test_split: test
cache_fields: false
split_model_for_images: false
depth_from_visual_hull: false
depth_range: [0, 2.4]
ignore_image_idx: []
model:
decoder: simple
encoder: resnet18
decoder_kwargs: {}
encoder_kwargs: {}
depth_function_kwargs: {}
c_dim: 256
lambda_occupied: 1.
lambda_freespace: 1.
lambda_rgb: 1.
lambda_depth: 0.
lambda_image_gradients: 0.
lambda_sparse_depth: 0.
lambda_normal: 0.05
reduction_method: sum
patch_size: 1
training:
out_dir: out/00
batch_size: 2
batch_size_val: 1
print_every: 10
checkpoint_every: 200
visualize_every: 10000
validate_every: 2000
backup_every: 50000
learning_rate: 0.0001
scheduler_milestones: [750, 1000]
scheduler_gamma: 0.5
model_selection_metric: loss
model_selection_mode: minimize
n_training_points: 4
n_eval_points: 4
n_workers: 0
logfile: train.log
sample_continuous: False
overwrite_visualization: true
occupancy_random_normal: False
depth_loss_on_world_points: False
use_cube_intersection: true
always_freespace: True
multi_gpu: false
test:
eval_file_name: eval_meshes
threshold: 0.5
model_file: model_best.pt
generation:
batch_size: 1
vis_n_outputs: 30
generation_dir: generation
simplify_nfaces: null
padding: 0.0
resolution_0: 32
upsampling_steps: 3
refinement_step: 30
refine_max_faces: 10000
with_colors: true
mesh_extension: ply
rendering:
render_dir: rendering
colors: rgb
resolution: [256, 256]
ray_sampling_accuracy: [1024, 1025]
extension: jpg
n_views: 1
n_start_view: 0
background: white

1. "images" contain multi-view images (png)
2. "mask" contain corresponding masks of these images (png)
3. "depth" contain corresponding depth of these images (png)
4. "cameras.npz" contain require .mat files

So, Now I wanna know how to train this pipeline for single RGB images for different classes so that we can generate 3D model from a single image only.
What changes need to be done in config file?
Or Mention any other changes need to be done.

from differentiable_volumetric_rendering.

Please refer the below config file also and updates the parameters.
https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/single_view_reconstruction/single_view_supervision/ours_combined.yaml

from differentiable_volumetric_rendering.

Once you have updated config file, then follow the instructions from here.
https://github.com/autonomousvision/differentiable_volumetric_rendering#training

from differentiable_volumetric_rendering.

After training the model, then you can refer the below section for testing.
https://github.com/autonomousvision/differentiable_volumetric_rendering#generation-from-your-own-single-images

from differentiable_volumetric_rendering.

from differentiable_volumetric_rendering.

Hi Nitish,
I tried with this config https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/single_view_reconstruction/single_view_supervision/ours_combined.yaml
But I'm facing many KeyError as train.py expect all the keys present in this config https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/default.yaml

Traceback (most recent call last): File "train.py", line 61, in <module> train_dataset = config.get_dataset(cfg, mode='train') File "/home/ujjawal/my_work/object_recon/differentiable_volumetric_rendering/im2mesh/config.py", line 151, in get_dataset method = cfg['method'] KeyError: 'method'

I tried adding the method parameter in the config file , but after this , again i got error for cachefields and so this continues for other parameters also .
How to deal with this?

from differentiable_volumetric_rendering.