nileshkulkarni / csm Goto Github PK

interesting

Thank you for your excellent work!

I have tried to run experiment/csm/csp in Colab, but it shows this error:

Do you have any idea?

I look forward to your answer.

Hi, it seems the training instructions doesn't provide information about how to prepare the Pascal 3D and ImageNet class dataset, could you please add some information? Thanks.

After PCK testing, we only have these mats:

How can we get this

as the APK Plots required?

Hi,
As part of a college project, we implemented CSM using the latest packages and renderers.
While we were able to generate visually good looking UV mappings the Percentage Correct Key Points metric values for the categories bird and car are only at 10% for the alpha 0.1.
We were wondering if there were any training tricks used by the author to get the number up to the 50s.
Also is it possible to provide the trained model dictionary at least for the bird or car category?. So that we can test if its the model that's causing the low PCK or something else.

Thanks

Eval keypoint transfer on birds throws error:

IOError: [Errno 2] No such file or directory: '/nfs.yoda/nileshk/CorrespNet/datasets/cubs/images/083.White_breasted_Kingfisher/White_Breasted_Kingfisher_0115_73252.jpg'

Eval keypoint transfer on cars throws error:

IOError: [Errno 2] No such file or directory: '/nfs.yoda/nileshk/CorrespNet/datasets/PASCAL3D+_release1.1/Images/car_imagenet/n04037443_13265.JPEG'

An extra null pointer check is not needed in functions like the following.

• DELETEARRAY
• GW_DELETE
• Matrix<T>::destroy

Hi, I get this error for running the "Eval keypoint transfer" commands.

File "csm/utils/render_utils.py", line 1, in
import pymesh
ImportError: No module named pymesh

I tried to install Pymesh but it only supports Python 3.7 - in your requirements it says Python 2.7 though. What do you suggest?

Hi, thanks for open-sourcing this wonderful work. Could you also provide weights of pretrained CSM model? Thanks!

I want to implement GCC loss in my project and I'm trying to figure out the details of your implementation.

Question-0: the branch predicting uv for each foreground pixel (used for GCC loss) is only used during training. So it is only used to indirectly help train the 'camera pose' branch, and 'deformation' branch and won't be used during inference. Is my understanding correct?

Here are your relevant code and my understanding/questions of the implementation of GCC loss.

Step-1, Set ground truth 'self.codes_gt['xy_map']', with linspace function (easy to understand~):

def get_sample_grid(img_size):
  | x = torch.linspace(-1, 1, img_size[1]).view(1, -1).repeat(img_size[0],1)
  | y = torch.linspace(-1, 1, img_size[0]).view(-1, 1).repeat(1, img_size[1])
  | grid = torch.cat((x.unsqueeze(2), y.unsqueeze(2)), 2)
  | grid.unsqueeze(0)
  | return grid

self.grid = cub_parse.get_sample_grid(self.upsample_img_size).repeat( 1, 1, 1, 1).to(self.device)

grid = self.grid.repeat(b_size,1,1,1)
self.codes_gt['xy_map'] = grid

Step-2, map predicted UV for each pixel to 3d vertices location, get 'points3d':

def project_uv_to_3d(uv2points, uv_map):
  | B = uv_map.size(0)
  | H = uv_map.size(1)
  | W = uv_map.size(2)
  | uv_map_flatten = uv_map.view(-1, 2)
  | points3d = uv2points.forward(uv_map_flatten)
  | points3d = points3d.view(B, H*W, 3)
  | return points3d

self.uv2points = cub_parse.UVTo3D(self.mean_shape)
points3d = geom_utils.project_uv_to_3d(self.uv2points, codes_pred['uv_map'])

Question-1: Why do you map UV to 3D first using 'UVTo3D', then use 'project_uv_to_3d'? What's the graphics formula/theory behind these two functions?

Step-3, orthographic project points3d to image plane, get 'codes_pred['project_points']'

def project_3d_to_image(points3d, cam, offset_z):

  | projected_points = orthographic_proj_withz(points3d, cam, offset_z)
  | return projected_points

def orthographic_proj_withz(X, cam, offset_z=0.):
  | """
  | X: B x N x 3
  | cam: B x 7: [sc, tx, ty, quaternions]
  | Orth preserving the z.
  | """
  | quat = cam[:, -4:]
  | X_rot = quat_rotate(X, quat)
  | scale = cam[:, 0].contiguous().view(-1, 1, 1)
  | trans = cam[:, 1:3].contiguous().view(cam.size(0), 1, -1)
  |  
  | proj = scale * X_rot
  |  
  | proj_xy = proj[:, :, :2] + trans
  | proj_z = proj[:, :, 2, None] + offset_z
  |  
  | return torch.cat((proj_xy, proj_z), 2)

codes_pred['project_points_cam_pred'] = geom_utils.project_3d_to_image(points3d, codes_pred['cam'], self.offset_z)
codes_pred['project_points_cam_pred'] = codes_pred['project_points_cam_pred'][..., 0:2].view(self.codes_gt['xy_map'].size())
codes_pred['project_points'] = codes_pred['project_points_cam_pred']

Step-4: GCC L2 loss between 'codes_pred['project_points''], and 'self.codes_gt['xy_map']'.

Reprojection Loss
project_points = codes_pred['project_points']
  | if opts.ignore_mask_gcc:
  | reproject_loss = reproject_loss_l2(project_points, codes_gt['xy_map'], seg_mask*0+1)
  | else:
  | reproject_loss = reproject_loss_l2(project_points, codes_gt['xy_map'], seg_mask)

Question-2: why does the 'codes_pred['project_points']' obtained by 'orthographic_proj_withz' range from 0 to 255 (e.g. when input image size is 256x256) ? I think 'points3d' is already in this range, but how does step-2 make this happen?

Hello,
Great work on CSM.
So I had two questions:

1. How do I map a 2d image to UV param.
2. I followed your instructions to convert my template off file to sphere (matlab code) then UV param it using the pythone module as described here: https://github.com/nileshkulkarni/csm/blob/master/csm/docs/preprocess.md
   But i am not sure how to read the final output .mat file to map between UV to 3d triangles.
   I would also like to map an image of the same class to this UV param (my Q1)
   Any help on this will be greatly appreciated. Thanks !

Hi!

Can you please provide pre-processed mesh templates used?

best,
Anand

Hello!

I have doubt about your PCK metrics.
In your paper you write:

Is h and w here the original height and width of images or cropped size (256)?
Thank you!

Hey, installation.md refers to csm_root/csm/external/ which appears to be missing

Thank you for your wonderful work!
I have two questions about the code.
Q1. In mean shape, Giving a point on the UV map, how to find its corresponding face(triangle)?
Q2. Asking for more references about the calculation of φ.
I have got that the calculation of φ(the process from UV map to 3d coordinate) was referred to this paper.
Could you please show me more references, such as links and formulations, about the calculation of φ.
For example, I know these lines are calculating the barycentric coordinates, but I want to know the formulations behind these codes.
Thanks a lot!!

Hi,I was trying to run model training command python -m csm.experiments.csm.csp --name=csm_bird_net --n_data_workers=4 --dataset=cub --display_port=8094 --scale_bias=0.75 --warmup_pose_iter=2000, it report:

 from ._conv import register_converters as _register_converters
loading /media/lab601/000555B8000E726C/wuy/csm/csm/data/../cachedir/cub/data/train_cub_cleaned.mat
5964 images
Loading Mean shape from /media/lab601/000555B8000E726C/wuy/csm/csm/data/../cachedir/cub/../shapenet/bird/shape.mat
Visdom Env Name csm_bird_net_wpose
create web directory /media/lab601/000555B8000E726C/wuy/csm/csm/experiments/csm/../../cachedir/web...
Traceback (most recent call last):
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/media/lab601/000555B8000E726C/wuy/csm/csm/experiments/csm/csp.py", line 439, in <module>
    app.run(main)
  File "/home/lab601/.local/lib/python2.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/lab601/.local/lib/python2.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/media/lab601/000555B8000E726C/wuy/csm/csm/experiments/csm/csp.py", line 434, in main
    trainer.train()
  File "csm/nnutils/train_utils.py", line 181, in train
    self.forward()
  File "/media/lab601/000555B8000E726C/wuy/csm/csm/experiments/csm/csp.py", line 219, in forward
    codes_pred = self.model.forward(feed_dict)
  File "csm/nnutils/icn_net.py", line 330, in forward
    unet_output = self.unet_gen.forward(img)
  File "csm/nnutils/unet.py", line 61, in forward
    return self.model(input)
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "csm/nnutils/unet.py", line 111, in forward
    self.x_enc = self.down(x_inp)
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab601/anaconda2/envs/csm/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CuDNN error: CUDNN_STATUS_MAPPING_ERROR

What's the solution to it?It's the cuda version problem?

Hey, first off thanks for the great work; it's great seeing progress towards less supervision even in 3D computer vision.

I was setting up the repo in order to run some comparisons and faced some issues while creating the environment from csm.yml. I run into the following issue due to missing packages

> conda env create -f csm.yml
ResolvePackageNotFound:
  - bleach=1.5.0=py27_0
  - html5lib=0.9999999=py27_0
  - cudatoolkit=8.0=3

Upon further inspection, it seemed the packages were outdated. I changed the package versions to something still available in conda forge but even that didn't help. After a few more changes i.e. shifting a couple libraries under pip in csm.yml, I was blocked by multiple conflicts.

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package wheel conflicts for:
pip==18.1=py27_0 -> wheel
Package sqlite conflicts for:
qt==5.6.3=h8bf5577_3 -> sqlite[version='>=3.25.3,<4.0a0']
python==2.7.15=h9bab390_4 -> sqlite[version='>=3.25.2,<4.0a0']
Package six conflicts for:
tensorflow-gpu-base==1.4.1=py27h01caf0a_0 -> six[version='>=1.10.0']
mock==2.0.0=py27_0 -> six
protobuf==3.6.1=py27he6710b0_0 -> six

Would you have any suggestions for the above? Should I create my own environment and install the necessary packages by myself? Would really appreciate your help with this, cheers!

Thanks for your excellent work!
It seems that links to download annotations doesn't work?
https://people.eecs.berkeley.edu/~shubhtuls/cachedir/vpsKps/segkps.zip
https://cmu.box.com/s/goter3meyi7rssbh2rujg36dfwygnc3f

Could you please update them?
Thanks!

Hello Nilesh,

In the quaternion azimuth-elevation predictor, linear layer output is never used. Is that intended? On this line.

Thanks.

Thank you for your excellent work!

We get (Batchsize, 256,256,2) as prediction now.

Could you tell me how can we convert this to RGB (to get the colorful image just like your results)?

Thanks!

Hello,
I would like to know if I could get access to the images from ImageNet used in your project or at least the mapping of image URLs and names used in the annotation files.