yezhen17 / 3dioumatch-pvrcnn Goto Github PK

Hello,

I noticed that this line does not work as intended. The self.unlabeled_index_list is initialized incorrectly because the elements in self.sample_index_list are string, not int. Therefore the set subtraction doesn't remove the labeled items. Can you confirm this?

I encountered such a problem during training and don't know how to solve it. I did not pre-train and used pv_rcnn_ssl.yaml
Traceback (most recent call last): File "train.py", line 237, in <module> main() File "train.py", line 151, in main total_epochs=args.epochs File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/__init__.py", line 53, in build_dataloader logger=logger, File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/kitti/kitti_dataset_ssl.py", line 25, in __init__ dataset_cfg=dataset_cfg, class_names=class_names, training=training, root_path=root_path, logger=logger File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/dataset.py", line 32, in __init__ ) if self.training else None File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/augmentor/data_augmentor.py", line 26, in __init__ cur_augmentor = getattr(self, cur_cfg.NAME)(config=cur_cfg) File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/augmentor/data_augmentor.py", line 37, in gt_sampling logger=self.logger File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/augmentor/database_sampler.py", line 24, in __init__ [self.db_infos[cur_class].extend(infos[cur_class]) for cur_class in class_names] File "/content/drive/MyDrive/3DIoU/3DIoUMatch-PVRCNN/pcdet/datasets/augmentor/database_sampler.py", line 24, in <listcomp> [self.db_infos[cur_class].extend(infos[cur_class]) for cur_class in class_names] KeyError: 'Car'

when I run
python -m pcdet.datasets.kitti.kitti_dataset create_part_dbinfos /tools/cfgs/dataset_configs/kitti_dataset.yaml train_0.01_1
there have been two results of splitting the data under different machines. One is only processing 37 sets of data, and the other is processing 3769 sets of data. Which is right here? I understand that the group of 37 is correct, but why does the group of 3769 appear?

Hi @THU17cyz

I have trouble when i reproduces SSL with 0.02 data.

I modified the code to use batch_size 4 in one gpu.

Then, I pre-trained and trained model with 0.02 split data that uploaded in your repository.

In conclusion, I modified the batch size(16->4)
I am not modified other setting(i.e. epochs, learning rate)

I think pre-trained model looks well, but ssl trained model does not looks good.

The first table is my pre-trained model's results, and the second one is my ssl-trained model's results.
I report the accuracy of the last epoch model.

Performance of cyclist is improved, but pedestrian's is decreased.

I only modified the code like this.
kitti_dataset_ssl.py line: 46

    if self.training:
        all_train = len(self.kitti_infos)
        #self.unlabeled_index_list = list(set(list(range(all_train))) - set(self.sample_index_list))  # float()!!!
        self.unlabeled_index_list = list(set(list(range(all_train))) - set([int(i) for i in self.sample_index_list]))
        # print(self.unlabeled_index_list)
        self.unlabeled_kitti_infos = []

Remove labeled data index from unlabeled_index_list.

kitti_dataset_ssl.py line: 390

    if self.training:
        #index_unlabeled = np.random.choice(self.unlabeled_index_list, 1)[0]
        index_unlabeled = np.random.choice(len(self.unlabeled_kitti_infos), 1)[0]
        info_unlabeled = copy.deepcopy(self.unlabeled_kitti_infos[index_unlabeled])

        data_dict_unlabeled = self.get_item_single(info_unlabeled, no_db_sample=True)
        return [data_dict_labeled, data_dict_unlabeled]
    else:
        return data_dict_labeled

np.random.choice(self.unlabeled_index_list,1)[0] => np.random.choice(len(self.unlabeled_kitti_infos),1)[0]

In pv_rcnn_ssl.py line: 38

if self.training:
        batch_dict['mask'] = batch_dict['mask'][:, :1] #  modify by chenxyyyy #modified by Honggyu
        mask = batch_dict['mask'].view(-1)

I add batch_dict['mask'] = batch_dict['mask'][:, :1] according to #2 (also about pseudo_sem_scores is modified)

In pv_rcnn_ssl.py line: 215

         if not self.unlabeled_supervise_cls:
            loss_rpn_cls = loss_rpn_cls[labeled_mask, ...].mean() #modified by Honggyu
        else:
            loss_rpn_cls = loss_rpn_cls[labeled_mask, ...].mean() + loss_rpn_cls[unlabeled_mask, ...].mean() * self.unlabeled_weight #modified by Honggyu

        loss_rpn_box = loss_rpn_box[labeled_mask, ...].mean() + loss_rpn_box[unlabeled_mask, ...].mean() * self.unlabeled_weight #modified by Honggyu
        loss_point = loss_point[labeled_mask, ...].mean() #modified by Honggyu
        loss_rcnn_cls = loss_rcnn_cls[labeled_mask, ...].mean() #modified by Honggyu

        if not self.unlabeled_supervise_refine:
            loss_rcnn_box = loss_rcnn_box[labeled_mask, ...].mean() #modified by Honggyu
        else:
            loss_rcnn_box = loss_rcnn_box[labeled_mask, ...].mean() + loss_rcnn_box[unlabeled_mask, ...].mean() * self.unlabeled_weight #modified by Honggyu

        loss = loss_rpn_cls + loss_rpn_box + loss_point + loss_rcnn_cls + loss_rcnn_box
        tb_dict_ = {}
        for key in tb_dict.keys():
            if 'loss' in key:
                tb_dict_[key+"_labeled"] = tb_dict[key][labeled_mask, ...].mean() #modified by Honggyu
                tb_dict_[key + "_unlabeled"] = tb_dict[key][unlabeled_mask, ...].mean() #modified by Honggyu
            elif 'acc' in key:
                tb_dict_[key+"_labeled"] = tb_dict[key][labeled_mask, ...].mean() #modified by Honggyu
                tb_dict_[key + "_unlabeled"] = tb_dict[key][unlabeled_mask, ...].mean() #modified by Honggyu
            elif 'point_pos_num' in key:
                tb_dict_[key + "_labeled"] = tb_dict[key][labeled_mask, ...].mean() #modified by Honggyu
                tb_dict_[key + "_unlabeled"] = tb_dict[key][unlabeled_mask, ...].mean() #modified by Honggyu
            else:
                tb_dict_[key] = tb_dict[key]

        tb_dict_['pseudo_ious'] = torch.cat(pseudo_ious, dim=0).mean()
        tb_dict_['pseudo_accs'] = torch.cat(pseudo_accs, dim=0).mean()
        # tb_dict_['sem_score_fg'] = sem_score_fg.mean()
        # tb_dict_['sem_score_bg'] = sem_score_bg.mean()
        tb_dict_['sem_score_fg'] = torch.cat(pseudo_sem_score_fg, dim=0).mean() #modified by Honggyu
        tb_dict_['sem_score_bg'] = torch.cat(pseudo_sem_score_bg, dim=0).mean() #modified by Honggyu

I modified .sum() to .mean() to compute loss.

And i modified batch_size = 2 to batch_size = 4 according to #2

And in pv_rcnn_ssl_60.yaml,
OPTIMIZATION.BATCH_SIZE_PER_GPU = 2.

Questions

1. Is my modification valid?
2. In your paper, is all reported performance from the last epoch model or from the best accuracy model?
   If you chose the best accuracy model, how do you define? Averaging all 3d bounding box accuracy?

Additionally, the model's performance fluctuates too much. Is it because the amount of labeled data is too small?(1%, 2%). or Is it because the learning rate is too high? or is it because my batch_size is too small? What do you think about it.

Here is my tensorboard results. ( _teacher is performance of teacher network)

SSL Trained model 0.02_1 in your repo.

SSL Trained model 0.02_2 in your repo.

SSL Trained model 0.02_3 in your repo.

Hello, thank you for sharing your codes!

According to your paper, 3DIoUMatch seems to use same augmentation strategy with SESS.
In SESS, point sub-sampling is used for weak augmentation.
I wonder which part of your code contains sub-sampling logic.

Thank you

Hello,
I have a question about the supervise_mode setting.
What's the difference when I set supervised_mode == 0, supervised_mode == 1 and supervised_mode == 2?

                        # only for 100% label
                        if **self.supervise_mode >**= 1:
                            filter = iou_max > 0.3
                            asgn = asgn[filter]
                            batch_dict['gt_boxes'][ind, ...][:] = torch.zeros_like(batch_dict['gt_boxes'][ind, ...][:])
                            batch_dict['gt_boxes'][ind, ...][:len(asgn)] = ori_unlabeled_boxes[i, :].gather(dim=0, 
                          index=asgn.unsqueeze(-1).repeat(1, 8))

                            if self.supervise_mode == 2:
                                batch_dict['gt_boxes'][ind, ...][:len(asgn), 0:3] += 0.1 * torch.randn((len(asgn), 3), device=iou_max.device) * \
                                                                                     batch_dict['gt_boxes'][ind, ...][
                                                                                     :len(asgn), 3:6]
                                batch_dict['gt_boxes'][ind, ...][:len(asgn), 3:6] += 0.1 * torch.randn((len(asgn), 3), device=iou_max.device) * \
                                                                                     batch_dict['gt_boxes'][ind, ...][
                                                                                     :len(asgn), 3:6]

hi, i wonder know batch_dict['gt_boxes'][ind, ...][:, 0:7] and ori_unlabeled_boxes[i, :, 0:7] represent what.
for batch_dict['gt_boxes'][ind, ...][:, 0:7], it is a pseudo_box from teacher network.
but for ori_unlabeled_boxes[i, :, 0:7], does it mean a gt box or what?

  anchor_by_gt_overlap = iou3d_nms_utils.boxes_iou3d_gpu(
                        batch_dict['gt_boxes'][ind, ...][:, 0:7],
                        ori_unlabeled_boxes[i, :, 0:7])

in pv_rcnn_ssl.py line 149-152.

/home/ji/anaconda3/envs/qiuqiu/lib/python3.6/runpy.py:125: RuntimeWarning: 'pcdet.datasets.kitti.kitti_dataset' found in sys.modules after import of package 'pcdet.datasets.kitti', but prior to execution of 'pcdet.datasets.kitti.kitti_dataset'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
Traceback (most recent call last):
File "/home/ji/anaconda3/envs/qiuqiu/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/ji/anaconda3/envs/qiuqiu/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/kitti/kitti_dataset.py", line 517, in
part_split=sys.argv[3]
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/kitti/kitti_dataset.py", line 452, in create_part_dbinfos
dataset = KittiDataset(dataset_cfg=dataset_cfg, class_names=class_names, root_path=data_path, training=True)
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/kitti/kitti_dataset.py", line 23, in init
dataset_cfg=dataset_cfg, class_names=class_names, training=training, root_path=root_path, logger=logger
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/dataset.py", line 32, in init
) if self.training else None
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/augmentor/data_augmentor.py", line 26, in init
cur_augmentor = getattr(self, cur_cfg.NAME)(config=cur_cfg)
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/augmentor/data_augmentor.py", line 37, in gt_sampling
logger=self.logger
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/augmentor/database_sampler.py", line 24, in init
[self.db_infos[cur_class].extend(infos[cur_class]) for cur_class in class_names]
File "/data/ji/code/3DIoUMatch-PVRCNN-main/pcdet/datasets/augmentor/database_sampler.py", line 24, in
[self.db_infos[cur_class].extend(infos[cur_class]) for cur_class in class_names]
KeyError: 'Car'

Hello, what is your GPU configuration? I trained with 8 GTX 1080Ti GPUs which are same as OpenPCDet, yet running this project reported the following error:
RuntimeError: CUDA out of memory. Tried to allocate 216.00 MiB (GPU 4; 10.91 GiB total capacity; 6.64 GiB already allocated; 151.38 MiB free; 6.71 GiB reserved in total by PyTorch)
By the way, my torch version is 1.5.0 and CUDA version is 9.0.176.
link said that pv_rcnn model occupied about 8700-9100M memory usage on each GPU with batch_size=2.

Hello, thank you for your codebase!

I am troubled that which part is the 3diou filter. The following code applies 3diou threshold (0.5/0.25/0.25 for car/ped/cyc)， but it seems like class confidence filter.

pv_rcnn_ssl.py line 81
valid_inds = pseudo_score > conf_thresh.squeeze()

Hello, @THU17cyz

https://github.com/THU17cyz/3DIoUMatch-PVRCNN/blob/1c4c525b7d3642021d6b67bb6fc6da027959a8ac/pcdet/models/detectors/pv_rcnn_ssl.py#L112-L119

I think this part has error.
'ori_boxes[i]' should be revised to 'ori_boxes[inds]'

A loop started from 116,
diff = max_pseudo_box_num - ori_boxes[inds].shape[0]
new_box = torch.cat([ori_boxes[inds], torch.zeros((diff, 8), device=ori_boxes[inds].device)], dim=0)
new_boxes[inds] = new_box

Could you check this for me?

Hi, Thanks for your open-soured code!

I have a question about the iou threshold setting on kitti dataset. In the readme file, you said the IoU thresholds are set 0.5, 0.25, 0.25 for car, pedestrian, and cyclist, respectively. In the last revised paper, the thresholds are set 0.7, 0.5, 0.5 with the same mAP results. Which is correct?

I run the pretraining of pvrcnn on kitti 1% labeled data based on your split. However, I get much worse results than yours.
Here is the data info

INFO  Database filter by min points Car: 184 => 168
INFO  Database filter by min points Pedestrian: 24 => 24
INFO  Database filter by min points Cyclist: 11 => 11
INFO  Database filter by difficulty Car: 168 => 127
INFO  Database filter by difficulty Pedestrian: 24 => 23
INFO  Database filter by difficulty Cyclist: 11 => 8
INFO  Loading KITTI dataset
INFO  Total samples for KITTI dataset: 37

Here is the result of mAP with 40 recalls

Car [email protected], 0.70, 0.70:
bbox AP40:57.0614, 52.8482, 48.1710
bev  AP40:48.6558, 46.2114, 41.6134
3d   AP40:10.9533, 11.8981, 11.5772
aos  AP40:52.69, 47.91, 43.78
Car [email protected], 0.50, 0.50:
bbox AP40:57.0614, 52.8482, 48.1710
bev  AP40:92.2586, 84.3057, 76.9679
3d   AP40:84.4601, 76.7450, 69.8524
aos  AP40:52.69, 47.91, 43.78
Pedestrian [email protected], 0.50, 0.50:
bbox AP40:6.2376, 6.3730, 6.1160
bev  AP40:0.3610, 0.6379, 0.6917
3d   AP40:0.2123, 0.2935, 0.3298
aos  AP40:3.35, 3.41, 3.21
Pedestrian [email protected], 0.25, 0.25:
bbox AP40:6.2376, 6.3730, 6.1160
bev  AP40:4.9317, 5.4842, 5.4997
3d   AP40:3.4472, 4.0254, 4.0970
aos  AP40:3.35, 3.41, 3.21
Cyclist [email protected], 0.50, 0.50:
bbox AP40:5.6924, 3.8000, 3.9303
bev  AP40:2.6481, 1.6338, 1.3922
3d   AP40:1.9624, 1.0719, 1.0538
aos  AP40:3.76, 2.20, 2.34
Cyclist [email protected], 0.25, 0.25:
bbox AP40:5.6924, 3.8000, 3.9303
bev  AP40:5.9757, 4.1799, 3.8489
3d   AP40:5.9757, 4.1799, 3.8489
aos  AP40:3.76, 2.20, 2.34

Could you help me to solve this issue?

Hello, I'm confused about the performance you report in your paper of arxiv vision and README.
You indicate in your paper that the performance is based on an IoU of 0.7, 0.5, 0.5, but in README it is based on 0.5, 0.25, 0.25.
So which one is real one?
Looking forward to your reply.
Thanks! @THU17cyz

Hello, Thank you for your nice work.

I have a problem in data preprocessing.

When i ran the data preprocessing code with the below command,
python -m pcdet.datasets.kitti.kitti_dataset create_part_dbinfos tools/cfgs/dataset_configs/kitti_dataset.yaml split_0.01_1

But I got this error,

Then, I ran the data infos generator code from original "OpenPCDet" repository for generating "kitti_dbinfos_train.pkl"

python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

But, when i ran this code, "kitti_dbinfos_train_3712.pkl" is made.

How i can solve this problem? Could you help me?

Hello! @THU17cyz Thank you for open-sourcing your codebase.

I have successfully run your pretrain phase on KITTI. But I had a problem running train phase.

when I use batchsize 1, I met a problem on the below code

# pcdet/models/detectors/pv_rcnn_ssl.py  line:257
def update_global_step(self):
    self.global_step += 1
    alpha = 0.999
    # Use the true average until the exponential average is more correct
    alpha = min(1 - 1 / (self.global_step + 1), alpha)
    for ema_param, param in zip(self.pv_rcnn_ema.parameters(), self.pv_rcnn.parameters()):
        ema_param.data.mul_(alpha).add_(1 - alpha,  param.data)

It notices that tha add_ cannot take 2 params，So I chage the code ema_param.data.mul_(alpha).add_(1 - alpha, param.data) to ema_param.data.mul_(alpha).add_((1 - alpha) * param.data).

Then I run sucessful

But when I change the batchsize to 2、4 or others，I met the index error on below.

# pcdet/models/detectors/pv_rcnn_ssl.py  line:67
for ind in unlabeled_mask:
    pseudo_score = pred_dicts[ind]['pred_scores']
    pseudo_box = pred_dicts[ind]['pred_boxes']
    pseudo_label = pred_dicts[ind]['pred_labels']
    pseudo_sem_score = pred_dicts[ind]['pred_sem_scores']

    if len(pseudo_label) == 0:
        pseudo_boxes.append(pseudo_label.new_zeros((0, 8)).float())
        continue

It shows that the num in unlabeled_mask beyond the range of pred_dicts. for example, the unlabeled_mask is [2,3,6,7] but the pred_dicts size is 4, 6,7 is illegal.

I want to know if my change is correct, and how to solve the error when the batch size is large？

Looking forward to your reply

thanks for your amazing work! After i deploy the environment successfully, and in the terminal,i can python and import spconv and pcdet successfully.But when i want to run the code, I meet a problem"connot import name iou3d_nms_cuda",how can i solve it?

Hi! Yezhen Cong:

I tested the model which train with 0 unlabeled weight. The result indicated that the model perform better than 1 unlabeled weight.

Does 0 unlabeled weight means completely learn from labeled data ?

Thanks! I'm confused for a long time.

Hello, thank you for your codebase!
I am trouble that the final detection result number is consistent with the input number.

2022-04-14 14:04:05,520   INFO  recall_roi_0.3: 0.000000
2022-04-14 14:04:05,520   INFO  recall_rcnn_0.3: 0.386345
2022-04-14 14:04:05,520   INFO  recall_roi_0.5: 0.000000
2022-04-14 14:04:05,520   INFO  recall_rcnn_0.5: 0.223867
2022-04-14 14:04:05,520   INFO  recall_roi_0.7: 0.000000
2022-04-14 14:04:05,520   INFO  recall_rcnn_0.7: 0.059667
2022-04-14 14:04:05,523   INFO  Average predicted number of objects(3744 samples): 7.316
Traceback (most recent call last):
  File "train.py", line 237, in <module>
    main()
  File "train.py", line 230, in main
    dist_test=dist_train
  File "/tmpnfs/01_3d/02_pl_3d/3DIoUMatch-CaDDN_02/tools/test.py", line 121, in repeat_eval_ckpt
    result_dir=cur_result_dir, save_to_file=args.save_to_file
  File "/tmpnfs/01_3d/02_pl_3d/3DIoUMatch-CaDDN_02/tools/eval_utils/eval_utils.py", line 113, in eval_one_epoch
    output_path=final_output_dir
  File "/tmpnfs/01_3d/02_pl_3d/3DIoUMatch-CaDDN_02/pcdet/datasets/kitti/kitti_dataset_ssl_mono.py", line 435, in evaluation
    ap_result_str, ap_dict = kitti_eval.get_official_eval_result(eval_gt_annos, eval_det_annos, class_names)
  File "/tmpnfs/01_3d/02_pl_3d/3DIoUMatch-CaDDN_02/pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 675, in get_official_eval_result
    gt_annos, dt_annos, current_classes, min_overlaps, compute_aos, PR_detail_dict=PR_detail_dict)
  File "/tmpnfs/01_3d/02_pl_3d/3DIoUMatch-CaDDN_02/pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 588, in do_eval
    min_overlaps, compute_aos)
  File "/tmpnfs/01_3d/02_pl_3d/3DIoUMatch-CaDDN_02/pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 469, in eval_class
    assert len(gt_annos) == len(dt_annos)
AssertionError````

File "/3DIoUMatch-PVRCNN-main/pcdet/models/detectors/pv_rcnn_ssl.py", line 73, in forward
pseudo_score = pred_dicts[ind]['pred_scores']
IndexError: list index out of range

pv_rcnn_ssl.py,the unlabeled_mask is [2,3,6,7],
but pred_dicts only have four elements, pred_dicts[ind] may be out of range

Thank you for your great research about ssl for 3D detection.

As the title states, I have some questions about weak augmentation for teacher network.

My questions are what augmentations are adopted, and why do they need.

I wonder why weak augmentations are applied because I think the teacher network will be able to generate better pseudo labels if augmentation is not applied.

I know that time has passed since the publication of your paper, but I would like you to answer my question. Thank you.

Thanks for your great works!

In pv_rcnn_ssl.py,

line 53 # self.pv_rcnn_ema.eval() # Important! must be in train mode

why should pv_rcnn_ema model be train mode?

Some layer works different at train mode and eval mode(for example, dropout layer, batch norm layer)

Ｉrun this code on 8 V100 GPUs with setting "train_0.1_1" in ./data/kitti, but a different result was produced.

pretrain

Car [email protected], 0.70, 0.70:
bbox AP:90.4946, 87.8960, 85.9724
bev AP:89.1578, 84.3747, 78.1961
3d AP:86.5061, 73.5857, 67.6990
aos AP:90.14, 86.91, 84.47
Car [email protected], 0.70, 0.70:
bbox AP:95.7267, 89.3033, 86.2265
bev AP:91.8607, 85.0761, 80.2126
3d AP:87.7689, 73.7096, 68.5043
aos AP:95.30, 88.26, 84.71
Car [email protected], 0.50, 0.50:
bbox AP:90.4946, 87.8960, 85.9724
bev AP:90.5059, 88.8931, 87.6750
3d AP:90.5026, 88.7312, 87.1547
aos AP:90.14, 86.91, 84.47
Car [email protected], 0.50, 0.50:
bbox AP:95.7267, 89.3033, 86.2265
bev AP:95.8453, 91.9532, 89.1067
3d AP:95.7830, 91.7464, 88.7804
aos AP:95.30, 88.26, 84.71
Pedestrian [email protected], 0.50, 0.50:
bbox AP:29.3085, 29.3043, 30.2714
bev AP:12.1812, 9.6326, 10.1116
3d AP:7.7328, 7.7264, 7.3232
aos AP:18.38, 17.35, 17.85
Pedestrian [email protected], 0.50, 0.50:
bbox AP:28.3785, 28.7564, 28.5835
bev AP:9.0040, 8.4401, 8.6786
3d AP:6.9333, 6.4495, 6.4089
aos AP:17.72, 17.06, 16.89
Pedestrian [email protected], 0.25, 0.25:
bbox AP:29.3085, 29.3043, 30.2714
bev AP:31.7072, 32.7220, 31.6044
3d AP:31.6231, 32.6293, 31.4090
aos AP:18.38, 17.35, 17.85
Pedestrian [email protected], 0.25, 0.25:
bbox AP:28.3785, 28.7564, 28.5835
bev AP:32.5248, 31.4872, 32.0119
3d AP:32.4313, 31.3309, 31.1840
aos AP:17.72, 17.06, 16.89
Cyclist [email protected], 0.50, 0.50:
bbox AP:64.7605, 46.2948, 40.6833
bev AP:62.0956, 42.9335, 37.8255
3d AP:59.9777, 37.5589, 36.4040
aos AP:59.62, 41.62, 36.61
Cyclist [email protected], 0.50, 0.50:
bbox AP:65.6239, 44.3176, 40.4676
bev AP:62.5017, 39.4443, 35.8029
3d AP:59.0120, 36.5243, 34.0512
aos AP:60.14, 39.35, 35.69
Cyclist [email protected], 0.25, 0.25:
bbox AP:64.7605, 46.2948, 40.6833
bev AP:63.9723, 44.2477, 38.8835
3d AP:63.9723, 44.2477, 38.8835
aos AP:59.62, 41.62, 36.61
Cyclist [email protected], 0.25, 0.25:
bbox AP:65.6239, 44.3176, 40.4676
bev AP:65.8755, 41.9935, 38.2332
3d AP:65.8755, 41.9935, 38.2332
aos AP:60.14, 39.35, 35.69

2021-08-10 08:50:04,806 INFO Result is save to /mnt/lustre/qiaozhijian/3DIoUMatch-PVRCNN/output/cfgs/kitti_models/pv_rcnn/pretrain_0.01_1/eval/eval_with_train/epoch_80/val

SSL0.1

Car [email protected], 0.70, 0.70:
bbox AP:90.5695, 88.4722, 86.0222
bev AP:89.3722, 84.2885, 78.6736
3d AP:87.3749, 75.5330, 68.4084
aos AP:89.97, 87.47, 84.73
Car [email protected], 0.70, 0.70:
bbox AP:96.1003, 91.0679, 86.6432
bev AP:92.3642, 85.4952, 82.3534
3d AP:88.2552, 75.8021, 70.9365
aos AP:95.39, 89.96, 85.32
Car [email protected], 0.50, 0.50:
bbox AP:90.5695, 88.4722, 86.0222
bev AP:90.5336, 89.1459, 87.9113
3d AP:90.5336, 89.0457, 87.6114
aos AP:89.97, 87.47, 84.73
Car [email protected], 0.50, 0.50:
bbox AP:96.1003, 91.0679, 86.6432
bev AP:96.1210, 93.6768, 90.9530
3d AP:96.0771, 91.9605, 89.0258
aos AP:95.39, 89.96, 85.32
Pedestrian [email protected], 0.50, 0.50:
bbox AP:39.2333, 38.0626, 35.9256
bev AP:21.4124, 19.1046, 17.8556
3d AP:16.4727, 15.8462, 15.4276
aos AP:22.01, 20.68, 19.36
Pedestrian [email protected], 0.50, 0.50:
bbox AP:35.6882, 33.2466, 32.1499
bev AP:19.7468, 17.3824, 16.9250
3d AP:15.5564, 14.1068, 13.2984
aos AP:21.43, 19.83, 19.06
Pedestrian [email protected], 0.25, 0.25:
bbox AP:39.2333, 38.0626, 35.9256
bev AP:39.8468, 35.2082, 34.9080
3d AP:39.8210, 35.1217, 34.8599
aos AP:22.01, 20.68, 19.36
Pedestrian [email protected], 0.25, 0.25:
bbox AP:35.6882, 33.2466, 32.1499
bev AP:38.8205, 35.0158, 34.0431
3d AP:38.7950, 34.9411, 33.9429
aos AP:21.43, 19.83, 19.06
Cyclist [email protected], 0.50, 0.50:
bbox AP:80.4026, 54.9720, 53.7440
bev AP:74.4704, 51.5182, 46.6159
3d AP:72.5318, 50.1919, 45.4282
aos AP:70.34, 47.42, 45.78
Cyclist [email protected], 0.50, 0.50:
bbox AP:80.2445, 53.6214, 51.1826
bev AP:76.9033, 49.3606, 46.4228
3d AP:74.8045, 47.7018, 43.8428
aos AP:69.73, 45.59, 42.78
Cyclist [email protected], 0.25, 0.25:
bbox AP:80.4026, 54.9720, 53.7440
bev AP:79.3710, 53.2193, 51.5633
3d AP:79.3710, 53.2193, 51.5633
aos AP:70.34, 47.42, 45.78
Cyclist [email protected], 0.25, 0.25:
bbox AP:80.2445, 53.6214, 51.1826
bev AP:79.4559, 51.8018, 49.1041
3d AP:79.4559, 51.8018, 49.1041
aos AP:69.73, 45.59, 42.78

2021-08-10 10:17:21,777 INFO Result is save to /mnt/lustre/qiaozhijian/3DIoUMatch-PVRCNN/output/cfgs/kitti_models/pv_rcnn_ssl_60/train_0.01_1/eval/eval_with_train/epoch_60/val

Ｑuestion

On validation datasets, my results are:
Car 73.7096 75.8021 +2.0925
Ped 6.4495 14.1068 +7.6573
Cyc 36.5243 47.7018 +11.1775
The order of magnitude is a little bit different．Which setting did you experiment on? Could you please provide a trained model?

Hello, thanks for your great work!
I want to know the location of the code change for the ema training manner.
I find that the teacher model weight update (pv_rcnn_ema) is related to the following code, did I miss anything?
Looking forward to your reply! @THU17cyz

Hello, i have a simple question?
in pv_rcnn_ssl.py line 55

i am not sure what this code means? Does it means i need to uncomment it when training and annotating it when testing? or just a reminder aims to state the _ema should be in eval state

Hello! Thank you for open-sourcing your codebase.

I have tried to use the repository to first do the pre-training phase on KITTI.
Actually, first, the repository would not run automatically for me, so I had to make some changes:
https://github.com/THU17cyz/3DIoUMatch-PVRCNN/blob/b50d9fd932c42e6916d29ad7153f1d3ccbd55827/pcdet/datasets/kitti/kitti_dataset.py#L452
I changed to training=False (because otherwise, it kept looking for a "kitti_dbinfos_train.pkl", which for some reason I couldn't properly generate with the full split txt - it just gave me an empty list [])
Here:
https://github.com/THU17cyz/3DIoUMatch-PVRCNN/blob/b50d9fd932c42e6916d29ad7153f1d3ccbd55827/pcdet/datasets/kitti/kitti_dataset.py#L226
I added

include_info_idxs = None
if len(self.sample_id_list[0].split(" ")) == 2: # means it's a split
    # generate stuff to include
    include_info_idxs = [int(line.split(" ")[1]) for line in self.sample_id_list]

for k in range(len(infos)):
    if include_info_idxs is not None:
        if k not in include_info_idxs:
            continue

because there didn't seem to be any code built-in for filtering just the frames inside the smaller split.

With these changes, I tried training on the provided 0.01_1 and 0.01_2 splits, and got:
Split 1:

Car [email protected], 0.70, 0.70:
bbox AP:97.0166, 88.4264, 87.7184
bev  AP:89.7851, 85.1410, 79.3515
3d   AP:88.5321, 77.2608, 75.0451
aos  AP:96.56, 87.50, 86.56
Car [email protected], 0.70, 0.70:
bbox AP:98.1557, 91.4289, 89.1516
bev  AP:95.1296, 85.9902, 83.5684
3d   AP:91.6326, 79.0419, 74.8542
aos  AP:97.68, 90.40, 87.91
Car [email protected], 0.50, 0.50:
bbox AP:97.0166, 88.4264, 87.7184
bev  AP:97.0982, 88.7778, 88.4398
3d   AP:97.0132, 88.7039, 88.2761
aos  AP:96.56, 87.50, 86.56
Car [email protected], 0.50, 0.50:
bbox AP:98.1557, 91.4289, 89.1516
bev  AP:98.2354, 93.8260, 91.6486
3d   AP:98.1979, 93.5422, 91.4403
aos  AP:97.68, 90.40, 87.91
Pedestrian [email protected], 0.50, 0.50:
bbox AP:50.1803, 49.5963, 45.4304
bev  AP:42.3984, 37.5393, 35.5838
3d   AP:39.5851, 34.8857, 30.8286
aos  AP:39.63, 38.13, 35.09
Pedestrian [email protected], 0.50, 0.50:
bbox AP:49.2451, 46.8583, 44.1228
bev  AP:40.1931, 35.3439, 31.6750
3d   AP:36.0652, 31.3460, 28.2745
aos  AP:37.15, 34.31, 32.17
Pedestrian [email protected], 0.25, 0.25:
bbox AP:50.1803, 49.5963, 45.4304
bev  AP:59.9030, 54.7150, 53.2468
3d   AP:59.8010, 54.6809, 53.1366
aos  AP:39.63, 38.13, 35.09
Pedestrian [email protected], 0.25, 0.25:
bbox AP:49.2451, 46.8583, 44.1228
bev  AP:59.3406, 55.3669, 52.1828
3d   AP:59.2904, 55.2920, 51.0803
aos  AP:37.15, 34.31, 32.17
Cyclist [email protected], 0.50, 0.50:
bbox AP:55.6971, 39.5461, 38.2764
bev  AP:54.8198, 38.2224, 37.1284
3d   AP:53.1655, 37.4204, 36.4887
aos  AP:46.97, 32.58, 31.53
Cyclist [email protected], 0.50, 0.50:
bbox AP:55.8919, 37.8161, 33.9657
bev  AP:54.7186, 35.0099, 32.7953
3d   AP:53.0402, 34.0532, 31.7318
aos  AP:46.12, 30.03, 27.06
Cyclist [email protected], 0.25, 0.25:
bbox AP:55.6971, 39.5461, 38.2764
bev  AP:55.2914, 38.8157, 37.4016
3d   AP:55.2914, 38.8157, 37.4016
aos  AP:46.97, 32.58, 31.53
Cyclist [email protected], 0.25, 0.25:
bbox AP:55.8919, 37.8161, 33.9657
bev  AP:55.2210, 36.9877, 33.1595
3d   AP:55.2210, 36.9877, 33.1595
aos  AP:46.12, 30.03, 27.06

Split 2:

Car [email protected], 0.70, 0.70:
bbox AP:96.6914, 88.7559, 87.1686
bev  AP:89.6414, 85.7397, 79.2499
3d   AP:88.1025, 77.2806, 69.6538
aos  AP:95.96, 87.87, 86.05
Car [email protected], 0.70, 0.70:
bbox AP:98.1452, 91.6966, 87.1124
bev  AP:95.1446, 86.3751, 81.5480
3d   AP:91.1320, 77.4718, 72.7242
aos  AP:97.34, 90.72, 85.97
Car [email protected], 0.50, 0.50:
bbox AP:96.6914, 88.7559, 87.1686
bev  AP:97.0062, 89.1552, 88.0703
3d   AP:97.5818, 89.0250, 87.5697
aos  AP:95.96, 87.87, 86.05
Car [email protected], 0.50, 0.50:
bbox AP:98.1452, 91.6966, 87.1124
bev  AP:98.2979, 94.3327, 89.6986
3d   AP:98.4326, 92.2598, 89.3975
aos  AP:97.34, 90.72, 85.97
Pedestrian [email protected], 0.50, 0.50:
bbox AP:57.6434, 52.8282, 47.3798
bev  AP:50.7659, 45.5090, 43.2965
3d   AP:48.7103, 43.6313, 38.8455
aos  AP:44.33, 40.31, 36.23
Pedestrian [email protected], 0.50, 0.50:
bbox AP:55.4885, 50.9032, 47.0967
bev  AP:49.5267, 44.7794, 39.9836
3d   AP:46.8996, 41.1147, 37.1701
aos  AP:41.19, 37.13, 33.90
Pedestrian [email protected], 0.25, 0.25:
bbox AP:57.6434, 52.8282, 47.3798
bev  AP:60.1065, 58.1532, 52.5542
3d   AP:60.0202, 54.5100, 52.4791
aos  AP:44.33, 40.31, 36.23
Pedestrian [email protected], 0.25, 0.25:
bbox AP:55.4885, 50.9032, 47.0967
bev  AP:60.1045, 56.1739, 51.0589
3d   AP:60.0185, 55.0687, 49.9226
aos  AP:41.19, 37.13, 33.90
Cyclist [email protected], 0.50, 0.50:
bbox AP:37.7206, 34.5612, 33.7128
bev  AP:33.1583, 29.8266, 29.3790
3d   AP:31.5708, 28.8987, 28.5315
aos  AP:35.55, 32.47, 31.58
Cyclist [email protected], 0.50, 0.50:
bbox AP:34.7546, 32.2395, 30.0869
bev  AP:31.0673, 26.7222, 25.0152
3d   AP:28.4158, 25.6330, 23.0792
aos  AP:32.82, 29.64, 27.79
Cyclist [email protected], 0.25, 0.25:
bbox AP:37.7206, 34.5612, 33.7128
bev  AP:34.2568, 30.1642, 29.7915
3d   AP:34.2568, 30.1642, 29.7915
aos  AP:35.55, 32.47, 31.58
Cyclist [email protected], 0.25, 0.25:
bbox AP:34.7546, 32.2395, 30.0869
bev  AP:31.7888, 28.1894, 25.4120
3d   AP:31.7888, 28.1894, 25.4120
aos  AP:32.82, 29.64, 27.79

Which are actually quite comparable to the paper's results:

but not the revised results in the readme:

which are much lower (lower numbers, at looser IoU threshold) due to fixed ground-truth sampling

I was wondering if you had any ideas why this may be the case?
I didn't even generate the full ground truth database, and during training, I see the line:

INFO  Total samples for KITTI dataset: 37

So indeed I am training on the reduced split

作者您好，我想研究的是半监督学习在单目3D目标检测上的应用（主要是kitti数据集，单目模型选择monodle,gupnet），就类似于您的pvrcnn实验。想先搭建一个半监督的框架，整体就如同您论文里面的那样，学生-教师，ema等等，具体的阈值过滤先不管，但是苦于没有合适的代码，您说您的代码只适合pvrcnn，那可以进行局部修改让他适用于单目模型啊？
期待您的回复