akar43 / categoryshapes Goto Github PK

Code for "Category-Specific Object Reconstruction from a Single Image" CVPR 2015.

MATLAB 89.67% M 0.74% C 9.34% Shell 0.25%

categoryshapes's Introduction

Category-Specific Object Reconstruction from a Single Image (CVPR 2015)

Abhishek Kar, Shubham Tulsiani, Joao Carreira, Jitendra Malik

Datasets and Paths:

You will need PASCAL VOC 2012 and PASCAL 3D+ to run the code. Download them and change the paths in startup.m to reflect your paths.

Compilation:

Run compile.m. Compile the vlfeat library under external/

Data:

$ sh setup_data.sh

The above script will download data for the project, PASCAL VOC 2012 and PASCAL 3D+ and put it under data/. It will also download and extract vlfeat in the external/.

All the required data can be downloaded from here. Unzip in BASE_DIR/data where BASE_DIR is the root directory for the codebase.

The code needs data formatted as in data/pascalData with keypoints and segmentations (either as polygons or binary masks). It also needs keypoint names as in data/partNames and the train/val split in PASCAL as in data/pascalTrainValIds and metadata about keypoints as in data/voc_kp_metadata to align to a canonical frame (here the Pascal 3D frame) and ensure a right handed co-ordinate system.

Train Models:

mainTrain('car','debug',<parameter options>)

This will train basis shape models for a particular class (here 'car') with the experiment id 'debug' that you can use later to visualize/test/evaluate. Look at get_params.m for description of parameters.

Test Models:

jobID = mainTest('car','debug','withKps',<parameter options>)

This will test the model you trained above (with train id 'debug') on the validation set. The test id (here 'withKps') is to enable testing with the same trained model with different test settings (e.g. with or without keypoints, with different parameters, with or without optimizing scale, translation, rotation etc).

Visualizing Results:

visInferredShapes('car', jobID)

Visualize the results for the models above. jobID is returned by mainTest after execution. It is usually 'Test<trainId><testId>'. Use this id to perform all operations on the test data (evaluation/visualization etc).

visNRSFMmodel('car', jobID)

Visualize the result of NRSFM. Shows the predicted 3D keypoints and their convex hull.

visDeformations('car', trainId)

Visualize the learnt deformation bases. Note: use trainId instead of jobID here.

Evaluation:

mainEval('car',jobID);

This will run the mesh and depth map evaluation on the cached meshes and depth maps.

Cached files:

All the files are cached under in 'cachedir' which is usually ./cache. The cached files are named as follows:

shapeModelNRSFM.mat - Parameters for the trained NRSFM model. Also contains test_model which contains the estimated parameters on the test set
shapeModelOpt<id>.mat - Parameters for the trained basis shape models.
statesDir<id> - Directory containing the projection parameters from NRSFM and other info per instance.
inferredShapes<id> - Directory containing fitted basis shapes to instances, projection parameters after optimization etc.
meshes<id> - Directory containing meshes fitted to each instance. Each file contains 'faces' and 'vertices'. Can be viewed using the showMesh function
depthMap<id> - Directory containing meshes rendered into depth maps per instance.
sirfs<id> - Directory containing the outputs from SIRFS. Contains albedo, depth, normals, shading and illumination maps.

categoryshapes's People

Contributors

Stargazers

Watchers

categoryshapes's Issues

Reproducing article results

Hello,

I have a few questions regarding the reproduction of the results given in the CVPR article, Table 1 and Table 2 in particular

In the article you mention object subcategories, however the code posted here does not use subcategories. There is a file subtypeClusters.mat that seems to divide Pascal VOC 3D+ objects into clusters, but it is not used in the code.
In Section 4.1, it is said that the 3D models are tested and fit without test/train split; since the code seems to make a train/test split, should I simply merge these two splits before evaluation to reproduce the results of Table 1?
When I follow the README instructions and get to launch mainEval('car',jobID); I obtain Mean mesh error: 3.214. I see this is quite lower than the results given in the article for the mean mesh error of the car class with KP+Mask, which is 5.18, or 5.07 for in Table 2 in a train/test setting: am I missing something here?

Thanks

Cannot run the experiment without mask (no sds files)

Hi,

Thanks for releasing a neat code.
I am running some experiments and I am having difficulty testing SDS case you reported.
Here is the error I am getting

Unable to read file
'/{myfolders}/CategoryShapes/cache/aeroplane/SDSmasks50/2010_003127_2'.
No such file or directory.

Error in getTestStateFiles>(parfor body) (line 54)
        [sdsbbox, bboxmask] = thisSDSMask(sdsdir,voc_id);

Error in getTestStateFiles (line 10)
parfor instance = 1:length(model_3d.c)

Error in testBasisShapes (line 11)
getTestStateFiles(nrsfmTestModel, statesDirTest, params.opt.relaxInit);

Error in mainTest (line 36)
testBasisShapes(shapeModelOpt,nrsfmTestModel,jobID);

Here are the parameters I used to run the test.

jobId = mainTest(class,'debug','withKpsSDS', ...                          
    {'opt.rotationPredNum','1', ...                                       
    'opt.relaxInit','{''mask''}', ...                                     
    'opt.relaxOpt','{''scale'',''translation'',''rotation'',''kps''}', ...
    'opt.lambda','[1 1 200 0.2 0.2 5]','opt.truncatedLoss','1'});

I guess you might have forgotten to include the logic to build or release imperfect segmentation.
Could you please help?

Thank you.

Cannot run the test with pose prediction.

I wanted to run the test without keypoint and it seems like rotation prediction takes an important role here.
However, I cannot find in your codebase how you fed data.rotationPred to your data.
I'm guessing based on your paper that you used some sort of cnn pose predictor.
Could you please share your rotationPred?

I'm using the following code to run the test

mainTest(class,'debug','poseHyp1SDS',...
         {'opt.rotationPredNum','1',...
         'opt.relaxInit','{''scale'',''translation'',''rotation'',''kps'',''mask''}',...
         'opt.relaxOpt','{''scale'',''translation'',''rotation''}',...
         'opt.useOrigTri','1',...
         'opt.lambda','[1 1 200 0.2 0.2 5]',...
         'opt.truncatedLoss','1'});

Thank you again!

Sofa category

Hello, I have a question about the sofa category: it seems that the output of the Visual Hull in the default settings set has holes (I guess because there are so few images / different viewpoints in the training set) and as a result, the inferred shapes are bad looking... Do you have an idea what could be done to fix this? Did you obtain the sofa results with less viewpoint clusters than the default, or maybe after merge of the train/test split? Thanks !