vegeta2020 / se-ssd Goto Github PK

Hi,

I am getting following error:

File "train.py", line 128, in
main()
File "train.py", line 125, in main
train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate, logger=logger,)
File "/Code/SE-SSD/det3d/torchie/apis/train_sessd.py", line 328, in train_detector
trainer.run(data_loaders, cfg.workflow, cfg.total_epochs, local_rank=cfg.local_rank)
File "/Code/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 472, in run
epoch_runner(data_loaders[0], data_loaders[1], self.epoch, **kwargs)
File "/Code/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 333, in train
for i, data_batch in enumerate(data_loader):
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in next
data = self._next_data()
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
return self._process_data(data)
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
data.reraise()
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise
raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/miniconda3/envs/py36/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Code/SE-SSD/det3d/datasets/kitti/kitti.py", line 170, in getitem
return self.get_sensor_data(idx, with_gp=False)
File "/Code/SE-SSD/det3d/datasets/kitti/kitti.py", line 209, in get_sensor_data
data, _ = self.pipeline(res, info)
File "/Code/SE-SSD/det3d/datasets/pipelines/compose.py", line 23, in call
res, info = t(res, info)
File "/Code/SE-SSD/det3d/datasets/pipelines/preprocess.py", line 93, in call
targeted_class_names=self.class_names,
File "/Code/SE-SSD/det3d/core/sampler/sample_ops_v2.py", line 187, in sample_all
"points": np.concatenate(s_points_list, axis=0),
File "<array_function internals>", line 6, in concatenate
ValueError: need at least one array to concatenate

I think the problem is with data loading, but not sure how to fix it. I have verified the dataset paths, and configuration file also looks good.
Any hint/suggestion will be appreciated.

Thanks,
Pervaiz Khan.

Hi,
I have a question about training details.
I trained SE-SSD several times, but the performance is different from your paper.

In AP_40 with (0.7, 0.7, 0.7), 3d detection AP for easy, moderate, and hard sets are respectively about 93.XX, 84.XX, and 81.XX. with the CIA-SSD model provided in https://github.com/Vegeta2020/CIA-SSD, your default configuration (examples/second/configs/config.py), and the provided codes.
This AP seems similar to the performance of only consistency loss mode in table 3.

Cons loss ODIoU SA-DA Easy Moderate Hard
- - - 92.58 83.22 80.15
- - X 93.02 83.70 80.68
- X - 93.07 83.85 80.78
X - - 93.13 84.15 81.17
X X - 93.17 85.81 83.01
X X X 93.19 86.12 83.31

Please let me know if there are any missing details.

I want use odiou_3D loss to my own network, but why the loss computed by odious.py can be -inf? Besides, the prediction will be NaN after training very little time?

I would like to know how to understand

“soft targets from the teacher often have higher entropy, thus offering
more information [9] for the student to learn from”

In a classified task, I can understand that. Because hard target is a one hot vector, but soft is not, so it will provide more information.
But how can I understand the soft target in the detection task has more information?

Thanks for your work and project, I have trained a model with your method.

I want to deploy this model with tensorrt inference engine in embedded platform, the first thing to do is export pytorch model to onnx model, but there are many dictionary data structures and custom layers/operators in this project, how to modify input、output and network ?

Could you give me some ideas ? Thanks !

I execute command "python3 train.py" and get error below. I don't know how to solve it.

When i run python setup.py install and encountered the following problem：


Can you give me some tips to make it work?

Hi, everyone, can you reproduce the paper result on val dataset use this repo? I can't reproduce. I first follow Det3d's tutorial to install Det3d, and prepare the info data and reduced data, and then use se-ssd to prepare gtaug data.
To run the code, I change some code following others in other question, just change model_ema = build_detector(), other change is the path problem and some hard code batch_size, not the core code.
For pertained model, I use the model provided in CIA-SSD repo.
What's more, I use 2 v100 32g gpu, and batch_size = 1 for each gpu, As I can only run the code with batch_size = 1, I don't know why now, so the total batch_size is 2 with 2 v100 32g gpu.
For the result, after SE-SSD training, compared with pertained model, the evaluation result is generally the same, don't have the improvement in paper.
So I just wonder does anyone can reproduce the result? Can you point out where I'm wrong, thank you very much.
@Vegeta2020 ,@pigtigger, @WWW2323.

Great works! I got a question when reading paper:
how do you calculate IoU between Bp and Bg in ODIoU loss? Do you calculate axis-aligned IoU or rotated IoU (the backward seems complicated)?

Where did the training reaults save?

And can you provide the version of python, pytorch, cuda you used?

I succeed in training and testing with Kittidataset.
It works well. Thank you!

By the way, i have my custom dataset(Format is same with Kittidataset)
so i want to change the voxel generator range on config.py
I changed like this
range=[0, -40.0, -3.0, 70.4, 40.0, 1.0], -> range=[-52.8, -32.0, -3.0, 52.8, 32.0, 1.0],

But i got this error
RuntimeError: The size of tensor a (84480) must match the size of tensor b (70400) at non-singleton dimension 0
I know this error is caused by incorrect the number of voxel from the different range.
Is there any ways to train and test with like my different range without changing existing model architecture?
What should i change?

Really Nice Work! I get two quick questions.

1> is the scale-aware data augmentation also used for preprocessing inputs for both teacher/student networks?

2> If the answer for 1 is no, for the last two rows of table 3, what does removing SA-DA mean? Does it mean that you remove SA-DA for the student network's input? And now the student network's prediction is based on a global transformed point cloud (but not dropout / sparsified through SA-DA)? It is surprising that this still works quite well.

Thanks

When I executed the instruction python train.py, an error is reported:
Traceback (most recent call last):
File "train.py", line 14, in
from det3d import version
ImportError: cannot import name 'version'

I configured the environment according to the steps,

Hello, thanks for your work!
I cannot install your codebase for some unknown reasons(docker environment), so I just copy your code. As for ODIOU loss,
self.odiou_3d_loss(boxes1, boxes2, weights, batch_size=4)
It need to run this only one line for 7 minutes (KITTI, SECOND model)!

Besides, the result of the first iter is also Irregular (train from sketch):
odiou_loss[odiou_loss>0].shape=1162, weights[weights>0].shape==512, odiou_loss[weights>0].sum()==0).

So, is the calculation time of ODIOU loss normal? And is the iou_loss output right(cannot be zero)?

original_pp_mghead_syncbn_kitti.py is for pointpillars Net, right?

To use pointpillars, all i have to do is change this on train.py?
before : parser.add_argument("--config", default='../examples/second/configs/config.py', help="train config file path")
after : parser.add_argument("--config", default='../examples/point_pillars/configs/original_pp_mghead_syncbn_kitti.py', help="train config file path")

I think additional process must be needed.
Could you let know me how i can use pointpillars? What should i change?

Hi, I just wonder why sessd need so much memory, the model looks not so large, for v100 32g, I can only set batch size =1, when batch size = 2, error happen:
runtime error: cudnn error: CUDNN_STSTUS_NOT_SUPPORTED, this error may appear if you passed in a non-contiguous input.
I just follow other change model_ema = build_detector() to solve the multiprocessing problem.

Hi~, your work is very awesome. But when I use the trained model you provide (https://drive.google.com/file/d/1M2nP_bGpOy0Eo90xWFoTIUkjhdw30Pjs/view?usp=sharing) to evaluate val set (by running "python test.py"), I get the results as follow:
Evaluation official_AP_11: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:90.61, 89.47, 89.02
bev AP:89.94, 88.11, 87.44
3d AP:88.44, 79.07, 78.27
aos AP:90.51, 89.10, 88.46
Evaluation official_AP_40: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:96.27, 93.00, 90.56
bev AP:93.25, 89.64, 87.23
3d AP:89.73, 83.08, 80.57
aos AP:96.12, 92.55, 89.97

Is the trained model a checkpoint of CIA-SSD or SE-SSD? The results of SE-SSD you provide is as follow:
AP_11: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:98.72, 90.10, 89.57
bev AP:90.61, 88.76, 88.18
3d AP:90.21, 86.25, 79.22
aos AP:98.67, 89.86, 89.16
AP_40: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:99.57, 95.58, 93.16
bev AP:96.70, 92.15, 89.74
3d AP:93.75, 86.18, 83.50
aos AP:99.52, 95.28, 92.69

Can you provide a checkpoint of the above results? I can’t train SE-SSD because of many environmental issues. Thanks very much!

I have run train.py and then the following error has occurred.

OSError: /mnt/proj50/zhengwu/saved_model/KITTI/proj52/megvii/second/pre_trained_model_2/epoch_60.pth is not a checkpoint file

What is the pre-trained mode?
Does the self-training on SE-SSD use a pre-trained model as the initial values of teacher model?
If yes, how can I get or create the pre-trained model?

Hey,
thanks for sharing the interesting project.

When I tried training from scratch without CIA-SSD pre-trained model, I found during computing consistency loss, the valid number of student and teacher is too large, even if the score threshold is given (0.3 in the code). So the iou computation between "top_box_preds_stu" and "top_box_preds_tea" will cause "cuda out of memory" error...

Many thanks in advance

Best regards,
Tim

File "/home/sl/code/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 266, in batch_processor_inline
losses = model(example, is_ema=[False, output_ema], return_loss=True)
File "/home/sl/anaconda3/envs/sessd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/sl/code/SE-SSD/det3d/models/detectors/voxelnet_sessd.py", line 41, in forward
return self.bbox_head.loss(example, preds, is_ema[1])
File "/home/sl/code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 709, in loss
loss_ema = self.get_model_ema_loss(example, preds_ema)
File "/home/sl/code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 836, in get_model_ema_loss
cls_loss = self.loss_cls(cls_preds, cls_targets, weights=cls_weights)
File "/home/sl/anaconda3/envs/sessd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/sl/code/SE-SSD/det3d/models/losses/losses.py", line 405, in forward
p_t = (target_tensor * prediction_probabilities) + ((1 - target_tensor) * (1 - prediction_probabilities))
RuntimeError: expected backend CUDA and dtype Float but got backend CUDA and dtype Int

2080TI cuda 10 spconv==1.0 python3.6 3.7 torch1.1.0 都是这个错

Can the provided model reproduce KITTI val 86.25(11 positions) & 86.18(40 positions)?

Dear authors, can I please ask when do you plan to release the code? Thanks!

Edit: Sorry, I just noticed that you want to release the code just after the CVPR conference. Any chance that you will release the code earlier?

Thanks for your work and project！

When I was trainning model, there was an error as follows:

python3 -m torch.distributed.launch --nproc_per_node=4 train.py
True
True
True
True
/home/firefly/project/se-ssd/se-ssd_vegeta2020/SE-SSD/examples/second/configs/config.py
/home/firefly/project/se-ssd/se-ssd_vegeta2020/SE-SSD/examples/second/configs/config.py
/home/firefly/project/se-ssd/se-ssd_vegeta2020/SE-SSD/examples/second/configs/config.py
/home/firefly/project/se-ssd/se-ssd_vegeta2020/SE-SSD/examples/second/configs/config.py
2021-07-20 06:57:03,695 - INFO - Distributed training: True
2021-07-20 06:57:03,695 - INFO - torch.backends.cudnn.benchmark: False
2021-07-20 06:57:03,746 - INFO - Finish RPN Initialization
2021-07-20 06:57:03,747 - INFO - num_classes: [1], num_preds: [14], num_dirs: [4]
2021-07-20 06:57:03,748 - INFO - Finish MultiGroupHead Initialization
2021-07-20 06:57:08,602 - INFO - {'Car': 5}
2021-07-20 06:57:08,603 - INFO - [-1]
2021-07-20 06:57:08,675 - INFO - load 2207 Pedestrian database infos
2021-07-20 06:57:08,675 - INFO - load 14357 Car database infos
2021-07-20 06:57:08,675 - INFO - load 734 Cyclist database infos
2021-07-20 06:57:08,676 - INFO - load 1297 Van database infos
2021-07-20 06:57:08,676 - INFO - load 488 Truck database infos
2021-07-20 06:57:08,676 - INFO - load 224 Tram database infos
2021-07-20 06:57:08,676 - INFO - load 337 Misc database infos
2021-07-20 06:57:08,676 - INFO - load 56 Person_sitting database infos
2021-07-20 06:57:08,706 - INFO - After filter database:
2021-07-20 06:57:08,706 - INFO - load 2104 Pedestrian database infos
2021-07-20 06:57:08,706 - INFO - load 10520 Car database infos
2021-07-20 06:57:08,706 - INFO - load 594 Cyclist database infos
2021-07-20 06:57:08,706 - INFO - load 826 Van database infos
2021-07-20 06:57:08,706 - INFO - load 321 Truck database infos
2021-07-20 06:57:08,706 - INFO - load 199 Tram database infos
2021-07-20 06:57:08,706 - INFO - load 259 Misc database infos
2021-07-20 06:57:08,706 - INFO - load 53 Person_sitting database infos
2021-07-20 06:57:08,825 - INFO - {'Car': 5}

......
......

~/se-ssd_vegeta2020/SE-SSD/det3d/torchie/apis/train_sessd.py", line 301, in train_detector
model_ema = copy.deepcopy(model)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 306, in _reconstruct
value = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 306, in _reconstruct
value = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 235, in
main()
File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 231, in main
cmd=process.args)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'train.py', '--local_rank=0']' returned non-zero exit status 1.

How to solve this problem? Could you give me some tips? Thanks!

When I run python setup.py build develop I get the following output:

Installed /home/user/Dokumente/SSDs/SE-SSD
Processing dependencies for det3d==1.0rc0+b68068d
error: protobuf 3.6.1 is installed but protobuf>=3.8.0 is required by {'tensorboardX'}

I think protobuf was installed with ros. How can I solve this problem? I already tried to upgrade it:

with pip show protobuf I get

Name: protobuf
Version: 3.17.3
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: None
Author-email: None
License: 3-Clause BSD License
Location: /home/fatih/.local/lib/python3.8/site-packages
Requires: six
Required-by: tensorboardX, det3d

I only found the single target training config file, did you try to train the model for multi targets at the same time? Thanks!

First of all, thank you for your paper!
I tried to use your code, but encountered the following problem
from ifp import ifp_sample
ModuleNotFoundError: No module named 'ifp'

I can't find anything about 'ifp',Can you help me solve this problem?

Thanks for your work.
Which is the model using in the inference phase? Student or Teacher model?

Hi, I am interested in how is the ap of Ped/Cyclist on Kitti? Could you share that?

Hi! @Vegeta2020 Thank you for your work!

I want to train SE-SSD on three class at the same time. I have modified your code. But I met a problem, and don't know how to solve it.

When I train 3 class on kitti, the program will fall into the following error!

File "train.py", line 120, in <module>
    main()
  File "train.py", line 117, in main
    train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate, logger=logger,)
  File "/data/SE-SSD/det3d/torchie/apis/train_sessd.py", line 324, in train_detector
    trainer.run(data_loaders, cfg.workflow, cfg.total_epochs, local_rank=cfg.local_rank)
  File "/data/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 473, in run
    epoch_runner(data_loaders[0], data_loaders[1], self.epoch, **kwargs)
  File "/data/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 347, in train
    outputs = self.batch_processor_inline(self.model, self.model_ema, data_batch, consistency_weight, train_mode=True, **kwargs)
  File "/data/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 267, in batch_processor_inline
    losses = model(example, is_ema=[False, output_ema], return_loss=True)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/SE-SSD/det3d/models/detectors/voxelnet_sessd.py", line 41, in forward
    return self.bbox_head.loss(example, preds, is_ema[1])
  File "/data/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 709, in loss
    consistency_loss = self.consistency_loss(preds_dicts, preds_ema, example)
  File "/data/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 679, in consistency_loss
    box_consistency_loss, idx1, idx2, mask1, mask2 = self.nn_distance(top_box_preds_stu, top_box_preds_tea)
  File "/data/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 578, in nn_distance
    ans_iou = iou3d_utils.boxes_iou_bev_gpu(box1, box2)
RuntimeError: copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

I have located the location of the program error. It's the following function. It throwed the error message at the beginning of training.

# det3d/models/bbox_heads/mg_head_sessd.py:  line679

# center consistency loss
box_consistency_loss, idx1, idx2, mask1, mask2 = self.nn_distance(top_box_preds_stu, top_box_preds_tea)

It seems that the iou3d_utils.boxes_iou_bev_gpu in self.nn_distance throwed the error.

def nn_distance(self, box1, box2, iou_thres=0.7, return_loss='10'):
    """
        box1: (N,C) torch tensor;   box2: (M,C) torch tensor
    """
    import pdb;pdb.set_trace()
    ans_iou = iou3d_utils.boxes_iou_bev_gpu(box1, box2)

I don't know why this mistake happened. Looking forward yo your reply!

My config file is shown below：

import itertools
import logging
from pathlib import Path

from det3d.builder import build_box_coder
from det3d.utils.config_tool import get_downsample_factor

data_root_prefix = "/data/SE-SSD/datasets"
# norm_cfg = dict(type='SyncBN', eps=1e-3, momentum=0.01)
norm_cfg = None
my_paras = dict(
    batch_size=1,
    data_mode="train",        # "train" or "trainval": the set to train the model;
    enable_ssl=True,         # Ensure "False" in CIA-SSD training
    eval_training_set=False,  # True: eval on "data_mode" set; False: eval on validation set.[Ensure "False" in training; Switch in Testing]

    # unused
    enable_difficulty_level=False,
    remove_difficulty_points=False,  # act with neccessary condition: enable_difficulty_level=True.
    gt_random_drop=-1,
    data_aug_random_drop=-1,
    far_points_first=False,
    data_aug_with_context=-1,        # enlarged size for w and l in data aug.
    gt_aug_with_context=-1,
    gt_aug_similar_type=False,
    min_points_in_gt=-1,
    loss_iou=None,
)


tasks = [
    dict(num_class=1, class_names=["Car"]),
    dict(num_class=1, class_names=["Pedestrian"]),
    dict(num_class=1, class_names=["Cyclist"]),
]

class_names = list(itertools.chain(*[t["class_names"] for t in tasks]))

# training and testing settings
target_assigner = dict(
    type="iou",
    anchor_generators=[
        dict(
            type="anchor_generator_range",
            sizes=[1.6, 3.9, 1.56],
            anchor_ranges=[0, -40.0, -1.0, 70.4, 40.0, -1.0],
            rotations=[0, 1.57],
            matched_threshold=0.6,
            unmatched_threshold=0.45,
            class_name="Car",
        ),
        dict(
            type="anchor_generator_range",
            sizes=[0.6, 0.8, 1.73],
            anchor_ranges=[0, -40.0, -0.6, 70.4, 40.0, -0.6],
            rotations=[0, 1.57],
            matched_threshold=0.4,
            unmatched_threshold=0.2,
            class_name="Pedestrian",
        ),
        dict(
            type="anchor_generator_range",
            sizes=[0.6, 1.76, 1.73],
            anchor_ranges=[0, -40.0, -0.6, 70.4, 40.0, -0.6],
            rotations=[0, 1.57],
            matched_threshold=0.4,
            unmatched_threshold=0.2,
            class_name="Cyclist",
        ),
    ],
    sample_positive_fraction=-1,
    sample_size=512,
    region_similarity_calculator=dict(type="nearest_iou_similarity",),
    pos_area_threshold=-1,
    tasks=tasks,
)

box_coder = dict(
    type="ground_box3d_coder", n_dim=7, linear_dim=False, encode_angle_vector=False,
)

# model settings
model = dict(
    type="VoxelNet",
    pretrained=None,
    reader=dict(
        type="VoxelFeatureExtractorV3",
        # type='SimpleVoxel',
        num_input_features=4,
        norm_cfg=norm_cfg,
    ),
    backbone=dict(
        type="SpMiddleFHD", num_input_features=4, ds_factor=8, norm_cfg=norm_cfg,
    ),
    neck=dict(
        type="SSFA",
        layer_nums=[5,],
        ds_layer_strides=[1,],
        ds_num_filters=[128,],
        us_layer_strides=[1,],
        us_num_filters=[128,],
        num_input_features=128,
        norm_cfg=norm_cfg,
        logger=logging.getLogger("RPN"),
    ),
    bbox_head=dict(
        # type='RPNHead',
        type="MultiGroupHead",
        mode="3d",
        in_channels=sum([128,]),
        norm_cfg=norm_cfg,
        tasks=tasks,
        weights=[1,],
        box_coder=build_box_coder(box_coder),
        encode_background_as_zeros=True,
        loss_norm=dict(
            type="NormByNumPositives", pos_cls_weight=1.0, neg_cls_weight=1.0,
        ),
        loss_cls=dict(type="SigmoidFocalLoss", alpha=0.25, gamma=2.0, loss_weight=1.0,),
        use_sigmoid_score=True,
        loss_bbox=dict(
            type="WeightedSmoothL1Loss",
            sigma=3.0,
            code_weights=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
            codewise=True,
            loss_weight=2.0,
        ),
        encode_rad_error_by_sin=True,
        loss_aux=dict(
            type="WeightedSoftmaxClassificationLoss",
            name="direction_classifier",
            loss_weight=0.2,
        ),
        direction_offset=0.0,
    ),
)

assigner = dict(
    box_coder=box_coder,
    target_assigner=target_assigner,
    out_size_factor=get_downsample_factor(model),
    debug=False,
)

train_cfg = dict(assigner=assigner)

test_cfg = dict(
    nms=dict(
        use_rotate_nms=True,
        use_multi_class_nms=False,
        nms_pre_max_size=1000,
        nms_post_max_size=100,
        nms_iou_threshold=0.01,
    ),
    score_threshold=0.3,
    post_center_limit_range=[0, -40.0, -5.0, 70.4, 40.0, 5.0],
    max_per_img=100,
)

# dataset settings
dataset_type = "KittiDataset"
data_root = data_root_prefix + "/KITTI/"
train_anno = data_root_prefix + "/KITTI/kitti_infos_train.pkl"
val_anno = data_root_prefix + "/KITTI/kitti_infos_val.pkl"
test_anno = data_root_prefix + "/KITTI/kitti_infos_test.pkl"
trainval_anno = data_root_prefix + "/KITTI/kitti_infos_trainval.pkl"


db_sampler = dict(
    type="GT-AUG",
    enable=True,
    db_info_path=data_root_prefix + "/KITTI/dbinfos_train.pkl",
    sample_groups=[dict(Car=15), dict(Pedestrian=8), dict(Cyclist=8),],
    db_prep_steps=[
        dict(filter_by_min_num_points=dict(Car=5, Pedestrian=5, Cyclist=5)),
        dict(filter_by_difficulty=[-1],),
    ],
    global_random_rotation_range_per_object=[0, 0],
    rate=1.0,
    gt_random_drop=my_paras['gt_random_drop'],
    gt_aug_with_context=my_paras['gt_aug_with_context'],
    gt_aug_similar_type=my_paras['gt_aug_similar_type'],
)
train_preprocessor = dict(
    mode="train",
    shuffle_points=True,
    gt_loc_noise=[1.0, 1.0, 0.5],
    gt_rot_noise=[-0.785, 0.785],
    global_rot_noise=[-0.785, 0.785],
    global_scale_noise=[0.95, 1.05],
    global_rot_per_obj_range=[0, 0],
    global_trans_noise=[0.0, 0.0, 0.0],
    remove_points_after_sample=True,
    gt_drop_percentage=0.0,
    gt_drop_max_keep_points=15,
    remove_unknown_examples=False,
    remove_environment=False,
    db_sampler=db_sampler,
    class_names=class_names,
)

val_preprocessor = dict(
    mode="val",
    shuffle_points=False,
    remove_environment=False,
    remove_unknown_examples=False,
)

voxel_generator = dict(
    range=[0, -40.0, -3.0, 70.4, 40.0, 1.0],
    voxel_size=[0.05, 0.05, 0.1],
    max_points_in_voxel=5,
    max_voxel_num=40000,
    far_points_first=my_paras['far_points_first'],
)

train_pipeline = [
    dict(type="LoadPointCloudFromFile"),
    dict(type="LoadPointCloudAnnotations", with_bbox=True),
    dict(type="Preprocess", cfg=train_preprocessor),
    dict(type="Voxelization", cfg=voxel_generator),
    dict(type="AssignTarget", cfg=train_cfg["assigner"]),
    dict(type="Reformat"),
    # dict(type='PointCloudCollect', keys=['points', 'voxels', 'annotations', 'calib']),
]
test_pipeline = [
    dict(type="LoadPointCloudFromFile"),
    dict(type="LoadPointCloudAnnotations", with_bbox=True),
    dict(type="Preprocess", cfg=val_preprocessor),
    dict(type="Voxelization", cfg=voxel_generator),
    dict(type="AssignTarget", cfg=train_cfg["assigner"]),
    dict(type="Reformat"),
]



data = dict(
    samples_per_gpu=my_paras['batch_size'],   # batch_size
    workers_per_gpu=1,
    train=dict(
        type=dataset_type,
        root_path=data_root,
        info_path=data_root + "/kitti_infos_train.pkl",
        ann_file=train_anno,
        class_names=class_names,
        pipeline=train_pipeline,
    ),
    val=dict(
        type=dataset_type,
        root_path=data_root,
        info_path=data_root + "/kitti_infos_val.pkl",
        ann_file=val_anno,
        class_names=class_names,
        pipeline=test_pipeline,
    ),
    test=dict(
        type=dataset_type,
        root_path=data_root,
        info_path=test_anno,
        ann_file=test_anno,
        class_names=class_names,
        pipeline=test_pipeline,
    ),
    train_unlabel_val=dict(
        type=dataset_type,
        root_path=data_root,
        info_path=val_anno,
        class_names=class_names,
        pipeline=train_pipeline,
        labeled=False,
    ),
    train_unlabel_test=dict(
        type=dataset_type,
        root_path=data_root,
        info_path=test_anno,
        class_names=class_names,
        pipeline=train_pipeline,
        labeled=False,
    ),
)

# optimizer
optimizer = dict(
    type="adam", amsgrad=0.0, wd=0.01, fixed_wd=True, moving_average=False,
)
"""training hooks """
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
# learning policy in training hooks
lr_config = dict(
    type="one_cycle", lr_max=0.003, moms=[0.95, 0.85], div_factor=10.0, pct_start=0.4,
)

checkpoint_config = dict(interval=1)  # 几个epoch保存一次
# yapf:disable
log_config = dict(
    interval=20,
    hooks=[
        dict(type="TextLoggerHook"),
        # dict(type='TensorboardLoggerHook')
    ],
)


# yapf:enable
# runtime settings
TAG = 'se-ssd-3class'
total_epochs = 300
device_ids = range(8)
dist_params = dict(backend="nccl", init_method="env://")
log_level = "INFO"
work_dir =  "/data/SE-SSD/datasets/" + TAG
load_from = None  #预训练
resume_from = None
workflow = [("train", 5), ("val", 1)]
save_file = False if TAG == "debug" or TAG == "exp_debug" or Path(work_dir, "Det3D").is_dir() else True

I would like to know how the pretrained weight(load file) was made.

Is it made by using Kitti dataset simply or did you use other datasets or processes for initialization?

How did you make it?

The reason why i asking this is that i want to train without any weights initially.

Thank you.

I'm really appreciate you for the nice source code, I run test.py and encountered the following problem：

Can you give me some tips to make it work?

请问，是不是和spconv1.0不兼容啊？

Hi,

I am getting following issue while training the model for a single GPU:

Traceback (most recent call last):
File "train.py", line 118, in
main()
File "train.py", line 115, in main
train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate, logger=logger,)
File "/Code/SE-SSD/det3d/torchie/apis/train_sessd.py", line 301, in train_detector
model_ema = copy.deepcopy(model)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File /miniconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 306, in _reconstruct
value = deepcopy(value, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/miniconda3/envs/py36/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects

Any help will be appreciated.
Thanks,

Hi, I have a question about test function. If I want to test with my own unlabeled data, what should I do?

When I eval in the val dataset, I got the result.
Evaluation official: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:84.92, 83.72, 83.56
bev AP:83.67, 81.57, 75.99
3d AP:79.95, 71.92, 71.52
aos AP:84.66, 83.21, 82.79
car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:84.92, 83.72, 83.56
bev AP:85.79, 84.89, 84.54
3d AP:85.48, 84.65, 84.30
aos AP:84.66, 83.21, 82.79

Evaluation coco: car coco [email protected]:0.05:0.95:
bbox AP:65.71, 63.42, 62.74
bev AP:65.17, 62.46, 61.28
3d AP:56.02, 52.30, 51.71
aos AP:65.48, 63.04, 62.18

It is lower than the result of paper. Can you give me some advice?

Thank you for sharing the code.

I am trying to train and test.

But i had a problem while training and testing with KITTI.

The issue is like this.

What is wrong with me?

If anyone knows what error it is, please let me know how to solve it.

Thank you.

Are the teacher and student SSD trained with anchors?

Will you release the code these days after the conference?

Traing kitti data, when i modify the config file from single car-task to all classes(car\pedestrian\cyclist)task, error occures.
The information is here:

Traceback (most recent call last):
File "train.py", line 118, in
main()
File "train.py", line 115, in main
train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate, logger=logger,)
File "/home/l4/wangmeng/SE-SSD/det3d/torchie/apis/train_sessd.py", line 321, in train_detector
trainer.load_checkpoint_from_scratch(cfg.load_from)
File "/home/l4/wangmeng/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 216, in load_checkpoint_from_scratch
self.optimizer.load_state_dict(checkpoint_0["optimizer"])
File "/home/l4/anaconda3/envs/SE-SSD/lib/python3.7/site-packages/torch/optim/optimizer.py", line 123, in load_state_dict
raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group

Dose anyone know the problem?

hi @Vegeta2020 , SE-SSD is a wonderful work ! but my results is so low after training about 50 epoch, I only change the data path and set load_from to the model provided in README , could you give me some advice?
Here is my results after training 50 epochs

File "/home/sl/code/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 346, in train
outputs = self.batch_processor_inline(self.model, self.model_ema, data_batch, consistency_weight, train_mode=True, **kwargs)
File "/home/sl/code/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 266, in batch_processor_inline
losses = model(example, is_ema=[False, output_ema], return_loss=True)
File "/home/sl/anaconda3/envs/SE-SSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/sl/code/SE-SSD/det3d/models/detectors/voxelnet_sessd.py", line 41, in forward
return self.bbox_head.loss(example, preds, is_ema[1])
File "/home/sl/code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 708, in loss
consistency_loss = self.consistency_loss(preds_dicts, preds_ema, example)
File "/home/sl/code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 678, in consistency_loss
box_consistency_loss, idx1, idx2, mask1, mask2 = self.nn_distance(top_box_preds_stu, top_box_preds_tea)
File "/home/sl/code/SE-SSD/det3d/models/bbox_heads/mg_head_sessd.py", line 578, in nn_distance
ans_iou = iou3d_utils.boxes_iou_bev_gpu(box1, box2)
File "/home/sl/code/SE-SSD/det3d/core/iou3d/iou3d_utils.py", line 50, in boxes_iou_bev_gpu
ans_iou = torch.cuda.FloatTensor(torch.Size((boxes_a.shape[0], boxes_b.shape[0]))).zero_()
RuntimeError: CUDA out of memory. Tried to allocate 18.19 GiB (GPU 0; 10.76 GiB total capacity; 2.85 GiB already allocated; 1.96 GiB free; 296.54 MiB cached)

I want to train without any weight files(load files).
But in order to train that, many cuda memories are needed(over 18 GiB).
I read that you have used a single TITAN Xp GPU, but i think it doesn't work on over 18 GiB things.
Did you initially train the model in parallel GPU or by using another GPU(like RTX 3090)?

and if i have enough CUDA memories, can i train the model normally?

Your response will be very helpful. Thank you.

Which one of the T-SSD and S-SSD will be used for the final inference?

It is mentioned in the right column of Page 3 that S-SSD is regarded as SE-SSD. But T-model is actually an ensemble of models?

Thanks for your explanation.

Traceback (most recent call last):
File "train.py", line 118, in
main()
File "train.py", line 115, in main
train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate, logger=logger,)
File "/home/muyangren907/cv/SE-SSD/det3d/torchie/apis/train_sessd.py", line 299, in train_detector
model_ema = copy.deepcopy(model)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 306, in _reconstruct
value = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects

Thank you for your released code.
I try to train model on 3-classes with provided pre-trained model, but there is an error " loaded state dict contains a parameter group that doesn't match the size of optimizer's group";
Without the pre-trained model, the error(runtimeError: CUDA out of memory. Tried to allocate 18.19 GiB (GPU 0; 10.76 GiB total capacity; 2.85 GiB already allocated; 1.96 GiB free; 296.54 MiB cached)) happend.
How can i solve the problem? Looking forward to your reply!

I want to get the txt for predicted result. I am looking forward to your reply!

For the proposed shape-aware data augmentation, where are the source code? Thanks!

Hello, when I want to build the iou3d,

I meet the following errors:

g++: error: /data/stan/SE-SSD/SE-SSD/det3d/core/iou3d/build/temp.linux-x86_64-3.6/src/iou3d.o: 没有那个文件或目录
g++: error: /data/stan/SE-SSD/SE-SSD/det3d/core/iou3d/build/temp.linux-x86_64-3.6/src/iou3d_kernel.o: 没有那个文件或目录
g++: error: /data/stan/SE-SSD/SE-SSD/det3d/core/iou3d/build/temp.linux-x86_64-3.6/src/iou3d_cpu.o: 没有那个文件或目录

where can I find these files?