All data used (datasets, models, results, ...) are stored in a directory local_data at the root of the repository. Create it with mkdir local_data or use a symlink if you want the data to be stored at a different place. We provide the utility cosypose/scripts/download.py for downloading required data and models. All of the files can also be downloaded manually.

For both T-LESS and YCB-Video, we use the datasets in the BOP format. If you already have them on your disk, place them in local_data/bop_datasets. Alternatively, you can download it using :

python -m cosypose.scripts.download --bop_dataset=ycbv
python -m cosypose.scripts.download --bop_dataset=tless

Additional files that contain information about the datasets used to fairly compare with prior works on both datasets.

python -m cosypose.scripts.download --bop_extra_files=ycbv
python -m cosypose.scripts.download --bop_extra_files=tless

We use pybullet for rendering images which requires object models to be provided in the URDF format. We provide converted URDF files, they can be downloaded using:

python -m cosypose.scripts.download --urdf_models=ycbv
python -m cosypose.scripts.download --urdf_models=tless.cad

In the BOP format, the YCB objects 002_master_chef_can and 040_large_marker are considered symmetric, but not by previous works such as PoseCNN, PVNet and DeepIM. To ensure a fair comparison (using ADD instead of ADD-S for ADD-(S) for these objects), these objects must not be considered symmetric in the evaluation. To keep the uniformity of the models format, we generate a set of YCB objects models_bop-compat_eval that can be used to fairly compare our approach against previous works. You can download them directly:

python -m cosypose.scripts.download --ycbv_compat_models

Notes:

• The URDF files were obtained using these commands (requires meshlab to be installed):

  python -m cosypose.scripts.convert_models_to_urdf --models=ycbv
  python -m cosypose.scripts.convert_models_to_urdf --models=tless.cad

• Compatibility models were obtained using the following script:

  python -m cosypose.scripts.make_ycbv_compat_models

The pre-trained models of the single-view pose estimator can be downloaded using:

# YCB-V Single-view refiner
python -m cosypose.scripts.download --model=ycbv-refiner-finetune--251020

# YCB-V Single-view refiner trained on synthetic data only 
# Only download this if you are interested in retraining the above model 
python -m cosypose.scripts.download --model=ycbv-refiner-syntonly--596719

# T-LESS coarse and refiner models 
python -m cosypose.scripts.download --model=tless-coarse--10219
python -m cosypose.scripts.download --model=tless-refiner--585928

To ensure a fair comparison with prior works on both datasets, we use the same detections as DeepIM (from PoseCNN) on YCB-Video and the same as Pix2pose (from a RetinaNet model) on T-LESS. Download the saved 2D detections for both datasets using

python -m cosypose.scripts.download --detections=ycbv_posecnn

# SiSo detections: 1 detection with highest per score per class per image on all images
# Available for each image of the T-LESS dataset (primesense sensor)
# These are the same detections as used in Pix2pose's experiments
python -m cosypose.scripts.download --detections=tless_pix2pose_retinanet_siso_top1

# ViVo detections: All detections for a subset of 1000 images of T-LESS.
# Used in our multi-view experiments.
python -m cosypose.scripts.download --detections=tless_pix2pose_retinanet_vivo_all

If you are interested in re-training a detector, please see the BOP 2020 section.

Notes:

• The PoseCNN detections (and coarse pose estimates) on YCB-Video were extracted and converted from these PoseCNN results.
• The Pix2pose detections were extracted using pix2pose's code. We used the detection model from their paper, see here. For the ViVo detections, their code was slightly modified. The code used to extract detections can be found here.

Training and evaluation code can be parallelized across multiple gpus and multiple machines using vanilla torch.distributed. This is done by simply starting multiple processes with the same arguments and assigning each process to a specific GPU via CUDA_VISIBLE_DEVICES. To run the processes on a local machine or on a SLUMR cluster, we use our own utility job-runner but other similar tools such as dask-jobqueue or submitit could be used. We provide instructions for single-node multi-gpu training, and for multi-gpu multi-node training on a SLURM cluster.

# CUDA ID of GPU you want to use
export CUDA_VISIBLE_DEVICES=0
python -m cosypose.scripts.example_multigpu

where scripts.example_multigpu can be replaced by scripts.run_pose_training or scripts.run_cosypose_eval (see below for usage of training/evaluation scripts).

Change the path to the code directory, anaconda location and specify a temporary directory for storing job logs by modifying `job-runner-config.yaml'. If you have access to a SLURM cluster, specify the name of the queue, it's specifications (number of GPUs/CPUs per node) and the flags you typically use in a slurm script. Once you are done, run:

runjob-config job-runner-config.yaml

# CUDA IDS of GPUs you want to use
export CUDA_VISIBLE_DEVICES=0,1
runjob --ngpus=2 --queue=local python -m cosypose.scripts.example_multigpu

The logs of the first process will be printed. You can check the logs of the other processes in the job directory.

runjob --ngpus=8 --queue=gpu_p1  python -m cosypose.scripts.example_multigpu

python -m cosypose.scripts.run_cosypose_eval --config ycbv

This will run the inference and evaluation on YCB-Video. We use our own implementation of the evaluation. We have checked that it matches the results from the original matlab implementation for the AUC of ADD-S and AUC of ADD(-S) metrics. For example, you can see that the PoseCNN results are similar to the ones reported in the PoseCNN/DeepIM paper:

PoseCNN/AUC of ADD(-S): 0.613

The YCB-Video results and metrics can be downloaded directly:

python -m cosypose.scripts.download --result_id=ycbv-n_views=1--5154971130

python -m cosypose.scripts.run_cosypose_eval --config tless-siso

This will run inference on the entire T-LESS dataset and print some metrics but not e_vsd<0.3 which is not supported in our code. The results can also be downloaded:

python -m cosypose.scripts.download --result_id=tless-siso-n_views=1--684390594

To measure e_vsd<0.3, we use the BOP Toolkit. You can run it using:

python -m cosypose.scripts.run_bop_eval --result_id=tless-siso-n_views=1--684390594 --method=pix2pose_detections/refiner/iteration=4

This will create a local_data/bop_predictions_csv/cosyposeXXXX-eccv2020_tless-test-primesense.csv file in the BOP format and run evaluation. Intermediate metrics and final scores are saved in local_data/bop_eval_outputs/cosposyXXXX-eccV2020_tless-test-primesense/, where XXXXX corresponds to a random number generated by the script.

The T-LESS SiSo results can also be downloaded directly:

python -m cosypose.scripts.download --bop_result_id=cosypose847205-eccv2020_tless-test-primesense

You can check the results match those from the paper:

cat local_data/bop_eval_outputs/cosypose847205-eccv2020_tless-test-primesense/error\=vsd_ntop\=1_delta\=15.000_tau\=20.000/scores_th\=0.300_min-visib\=0.100.json

{
  "gt_count": 69545,
  "mean_obj_recall": 0.6378486071644157,
  "mean_scene_recall": 0.6444110450903551,
  ...
  "recall": 0.632720209307857,
  ...
  "targets_count": 50452,
  "tp_count": 31922
}

Following other works, we reported mean_obj_recall in the paper.

You can visualize the single-view predictions using this notebook as example.

The pose estimation models are trained on a mix of real images provided with the T-LESS/YCB-Video datasets and a set of images that we generated. For each dataset, we generated 1 million synthetic images. You can download these large datasets:

# 106 GB
python -m cosypose.scripts.download --synt_dataset=tless-1M

# 113 GB
python -m cosypose.scripts.download --synt_dataset=ycbv-1M

We provide below the instructions to generate these dataset locally if you are interested in using our synthetic data generation code.

The synthetic training images are generated with some domain randomization. It includes adding textures to the background (and objects and T-LESS). We use a set of textures extracted from ShapeNet objects. Download the texture dataset:

python -m cosypose.scripts.download --texture_dataset

The synthetic images are generated using multiple processes managed by dask. The synthetic training images can be generated using the following commands for both datasets:

export CUDA_VISIBLE_DEVICES=0
python -m cosypose.scripts.run_dataset_recording --config tless --local
python -m cosypose.scripts.run_dataset_recording --config ycbv --local

Make sure that enough space is available on your disk. We generate 1 million images which is around 120GB for each dataset. Note that we use a high number of synthetic images, but it may be possible to use fewer images. Please see directly the script scripts/run_dataset_recording.py for additionnal parameters. It is also possible to use dask-jobqueue to generate the images on a cluster but we do not provide a simple configuration script for this at the moment. If you are interested in generating data using multiple machines on a cluster, you will have to modify dask-jobqueue's Cluster definition here.

You can visualize the images of the generated dataset using this notebook. You can check that the ground truth prvided by a dataset is correct using this notebook.

We apply data augmentation to the training images. Data augmentation includes pasting random images of the pascal VOC dataset on the background of the scenes. You can download Pascal VOC using the following commands:

cd local_data
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
tar -xvf VOCtrainval_11-May-2012.tar

(If the website is down, which happens periodically, you can alternatively download these files from a mirror at https://pjreddie.com/media/files/VOCtrainval_11-May-2012.tar)

Once you have generated the synthetic data images and downloaded pascal VOC, you can run the training script. On YCB-Video, we train a coarse model on synthetic data only and fine-tune it on the synthetic + real images. On T-LESS, we train a coarse and refinement model and synthetic + provided real images of isolated objects directly from scratch. In our experiments, all models are trained using the same procedure on 32 GPUs.

runjob --ngpus=32 python -m cosypose.scripts.run_pose_training --config ycbv-refiner-syntonly
runjob --ngpus=32 python -m cosypose.scripts.run_pose_training --config ycbv-refiner-finetune
runjob --ngpus=32 python -m cosypose.scripts.run_pose_training --config tless-coarse
runjob --ngpus=32 python -m cosypose.scripts.run_pose_training --config tless-refiner

You can visualize the logs of the provided models in this notebook.

You can add the run_id of each model that are your are training to visualize training metrics.

Notes:

• While we used 32 GPUs in our experiments, the training script can be ran with any number of GPUs. It will just be slower and the overall batch size will be smaller. We have not studied the impact of batch size on final performance of the model. On 32 NVIDIA V100, training a model takes approximately 10 hours. Note that the models are trained from scratch on all the objects of each dataset simulatenously.
• If you are interested in training with limited resources, you could consider the following changes to the code: (a) use a smaller backbone, e.g. flownet, resnet18 or resnet34, (b) train for fewer iterations, (c) start from one of our pre-trained models. All the parameters are defined in cosypose/scripts/run_pose_training.py. If you are trying to train with limited resources or on your own dataset and datas, please do not hesitate to share your experience, by opening an issue or by sending an email !
• We run evaluation of the models a few times during training. You can disable it by adding the flag --no-eval to speed up training. Note that the we do not use the evaluation metrics to find the best model since no official validation splits are available for YCB-Video/T-LESS. We always report results for the model obtained at the end of the training.

The following scripts will run the full CosyPose pipeline (single-view predictions + multi-view scene reconstruction), compute the metrics reported in the paper and save the results to a directory in local_data/results/.

export CUDA_VISIBLE_DEVICES=0
python -m cosypose.scripts.run_cosypose_eval --config tless-vivo --nviews=4
python -m cosypose.scripts.run_cosypose_eval --config tless-vivo --nviews=8
python -m cosypose.scripts.run_cosypose_eval --config ycbv --nviews=5

Note that the inference and evaluation can be sped up using runjob if you have access to multiple GPUs. The mAP@ADD-S<0.1d and AUC of ADD-S metrics are computed using our own code since they are not supported by the BOP toolkit. We refer to the appendix of the main paper for more details on these metrics.

The results can be also downloaded directly:

# YCB-Video 5 views
python -m cosypose.scripts.download --result_id=ycbv-n_views=5--8073381555 

# T-LESS ViVo 4 views
python -m cosypose.scripts.download --result_id=tless-vivo-n_views=4--2731943061

# T-LESS ViVo 8 views
python -m cosypose.scripts.download --result_id=tless-vivo-n_views=8--2322743008

On T-LESS ViVo, the evsd<0.3 and ADD-S<0.1d metrics are computed using the BOP toolkit, for example for computing the multi-view results for ViVo 8 views:

python -m cosypose.scripts.run_bop_eval  --results  tless-vivo-n_views=8--2322743008 --method pix2pose_detections/ba_output+all_cand --vivo

The ba_output+all_cand predictions correspond to the output of CosyPose concatenated to all the single-view candidates as explained in the experiment section of the paper. The single-view candidates have strictly lower score than the multi-view predictions, which means that single-view estimates are used for evaluation only if there are no multi-view predictions for an object, e.g. typically because a camera cannot be placed with respect to the scene because there are too few inlier candidates.

We also provide the BOP evaluation results that we computed and reported in the paper:

# T-LESS ViVo 1 view
python -m cosypose.scripts.download --bop_results=cosypose68486-eccv2020_tless-test-primesense

# T-LESS ViVo 4 views
python -m cosypose.scripts.download --bop_results=cosypose615294-eccv2020_tless-test-primesense

# T-LESS ViVo 8 views
python -m cosypose.scripts.download --bop_result_id=cosypose114533-eccv2020_tless-test-primesense

You can use this notebook to visualize the multi-view results on YCB-Video and T-LESS and generate the 3D visualization GIFs.

Stage 2 and 3 of CosyPose are agnostic to the 6D pose estimator used, and can therefore be combined with many other existing single-view object pose estimation method to solve problems on other datasets, or for real applications. We provide a utility for running CosyPose given a set of input 6D object candidates in each image.

If you are willing to combine CosyPose with your own pose estimator, you will need to provide the following:

• The 3D models of the objects considered and their associated symmetries. The models should be provided in a format similar to the BOP format in a models directory.
• A set of input 6D object candidates in each image candidates.csv. We use the same convention as the BOP format, but all the candidates in this file must be provided for a unique scene (a single 3D reconstruction) in different views.
• The intrinsics parameters of the cameras of each view in a file scene_camera.json following the BOP format.

Use these commands to create a custom scenario with T-LESS objects and run CosyPose on it:

cd local_data
mkdir -p custom_scenarios/example
ln -s $(pwd)/bop_datasets/tless/models custom_scenarios/example

export CUDA_VISIBLE_DEVICES=0
python -m cosypose.scripts.download --example_scenario
python -m cosypose.scripts.run_custom_scenario --scenario=example

This will generate the following files:

• results/subscene=0/predicted_scene.json a set of predicted objects and cameras with their associated poses in a common reference frame.
• results/subscene=0/scene_reprojected.csv poses of predicted objects expressed in camera frames, in the BOP format.

You can use this as an example to check the different formats in which the informations should be provided.

Notes:

• This is experimental. The default parameters for the pipeline should give good results in many scenarios (we use the same on YCB-Video and T-LESS) but we have yet not conducted experiments in many custom scenarios. If you are trying to apply CosyPose to your own 6D pose estimations and encounter any issues or would like to obtain better results, please consider sharing your experience, I would be very happy to help you.

• The script is quite slow to run for a single scene because all models need to be loaded and the first cuda call with pytorch is always slow. If you would like to use this for an application, consider using directly the API of the MultiviewScenePredictor in your own code. You can use the script scripts/run_custom_scenario.py as an example on how to use it.

• While the object candidate matching stage (stage 2 of CosyPose) is fairly optimized using a combination of C++ and fully batched operations in pytorch on GPU, the stage 3 (global scene refinement via object-level bundle adjustment) could largely be sped up. The implementation would benefit from being written in C++ with a standard optimization framework instead of using pytorch to compute full jacobians for Levenberg-Marquart. If you find this stage to be too slow for your problem/application, please open an issue or let me know. If there is demand for a faster implementation, I may provide it in the future.

We provide the training code that we used to train single-view single-object pose estimation models on the 7 core datasets (LM-O, TLESS, TUD-L, IC-BIN, ITODD, HB, YCB-V) and pre-trained detector and pose estimation models. Note that these models are different from the ones used in the paper. The differences with the models used in the paper are the following:

• In the paper, we use already available detectors for T-LESS and YCB-Video. For the BOP20 challenge, we trained our own detectors on each dataset.
• Detection and pose estimation models are trained using PBR synthetic images provided with the BOP challenge instead of using our own synthetic data to make it easier to compare fairly with the other approaches.
• In the BOP20 challenge results, the initialization of the pose provided to the coarse model is slightly different. First, the canonical orientation has been changed to have the z-axis parallel to the camera instead of having the x-axis parallel to the camera, a position with z-axis upward and parallel to the camera makes the overall shape and details of the objects more visible. Second, instead of fixing the z value of the canonical translation to 1 meter, we compute a guess of object depth using the height and width of the 2D bounding box and the 3D model. This makes the method more general as the canonical depth is always within a reasonable range of the correct depth even if the object is very far from the camera.

Even though the challenge is focused on single-view pose estimation, we also reported multi-view results on YCB-Video, T-LESS and HB for 4 and 8 views.

python -m cosypose.scripts.download --bop_dataset=DATASET --pbr_training_images
python -m cosypose.scripts.download --urdf_models=DATASET

for DATASET={hb,icbin,itodd,lm,lmo,tless,tudl,ycbv}. If you are not interested in training the models, you can remove the flag --pbr_training_images and you can omit lm.

You can download all the models that we trained for the challenge using our downloading script:

python -m cosypose.scripts.download --model=model_id

where model_id is given by the table below:

The detectors are MaskRCNN models with resnet50 FPN backbone. PBR corresponds to training only on provided synthetic images. SYNT+REAL corresponds to training on all available synthetic and real images when available (only for tless, tudl and ycbv). SYNT+REAL models are pre-trained from PBR.

If you want to use all the models for a complete evaluation:

python -m cosypose.scripts.download --all_bop20_models

The following commands will reproduce the results that we reported on the leaderboard for all the datasets:

# CosyPose-ECCV20-PBR-1VIEW
python -m cosypose.scripts.run_bop_inference --config bop-pbr

# CosyPose-ECCV20-SYNT+REAL-1VIEW
python -m cosypose.scripts.run_bop_inference --config bop-synt+real

# CosyPose-ECCV20-SYNT+REAL-1VIEW-ICP
python -m cosypose.scripts.run_bop_inference --config bop-synt+real --icp

# CosyPose-ECCV20-SYNT+REAL-4VIEWS
python -m cosypose.scripts.run_bop_inference --config bop-synt+real --nviews=4

# CosyPose-ECCV20-SYNT+REAL-8VIEWS
python -m cosypose.scripts.run_bop_inference --config bop-synt+real --nviews=8

The inference script is compatible with runjob.

Inference results on all datasets can be downloaded directly:

python -m cosypose.scripts.download --result_id=result_id

where result_id is given by the table below

If you want to download everything:

python -m cosypose.scripts.download --all_bop20_results

Notes:

• The ICP refiner was adapted from Pix2Pose code. Be careful if you want to use it, it slightly decrease performance over RGB-only on T-LESS instead of improving the results. Qualitative results show a misalignment of many objects after ICP, there is likely a small bug with my version but I haven't had time to go in detail. Note that our method and paper are focused on the RGB-only setting.

You can run locally the evaluation on the publicly available test sets:

python -m cosypose.scripts.run_bop20_eval_multi --result_id=result_id --method=method

where method is maskrcnn_detections/refiner/iteration=4 for single-view, maskrcnn_detections/icp when ICP is ran, and maskrcnn_detections/multiview for multi-view (n_views > 1).

If you are only interested in generating the bop predictions file suitable for submission to the website, you can run

python -m cosypose.scripts.run_bop20_eval_multi --result_id=result_id --method=method --convert_only

We use torchvision's MaskRCNN implementation for the detection. The models were trained using:

runjob --ngpus=32 python -m cosypose.scripts.run_detector_training --config bop-DATASET-TRAINING_IMAGES

where DATASET={lmo,tless,tudl,icbin,itodd,hb,ycbv} and TRAINING_IMAGES={pbr,synt+real} (synt+real only for datasets where real images are available: tless, tudl and ycbv).

runjob --ngpus=32 python -m cosypose.scripts.run_pose_training --config bop-DATASET-TRAINING_IMAGES-MODEL_TYPE

where MODEL_TYPE={coarse,refiner}.