kemo-huang / jmodt Goto Github PK

Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving

License: MIT License

Python 83.27% C++ 5.53% Cuda 10.68% C 0.52%

jmodt's Introduction

JMODT

This is the official code release of the IROS-2021 paper JMODT: Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving.

Overview

The system architecture of JMODT:

The region proposal feature processing modules:

Model Zoo

The results are evaluated on the validation set of the KITTI object tracking dataset. Only Car objects are used. The average precision (AP) scores are measured with 40 recall positions. The run time is only measured for the tracking part (after the region proposal feature processing).

Model	AP-Easy	AP-Moderate	AP-Hard	MOTA	MOTP	IDS	FRAG	Runtime
JMODT	94.01	87.37	85.22	86.10	87.13	0	129	0.01s

Requirement

The code has been tested in the following environment:

Ubuntu 20.04 & Windows 10
Python 3.8
PyTorch 1.9.0
CUDA Toolkit 11.1

Installation

Install PyTorch and CUDA.
Install other required Python packages:

pip install -r requirements.txt

Build and install the required CUDA modules via PyTorch and the CUDA toolkit:

python setup.py develop

Getting Started

Dataset preparation

Please download the official KITTI object tracking dataset.

To generate the detection results, please use the following command to reformat the ground truth to KITTI's object detection format. You can create your own data splits by modifying jmodt/config.py file.

python tools/kitti_converter.py --data_root ${DATA_ROOT}

The final dataset organization should be like this (you can have your custom data root):

JMODT
├── data
    ├── KITTI
        ├── tracking
        │   ├──training
        │   │  ├──calib & velodyne & label_02 & image_02
        │   ├──testing
        │      ├──calib & velodyne & image_02
        ├── tracking_object
            ├──ImageSets
            │  ├──small_val.txt & test.txt & train.txt & val.txt
            ├──training
            │  ├──calib & velodyne & label_2 & image_2 & sample2frame.txt & seq2sample.txt
            ├──testing
               ├──calib & velodyne & image_2 & sample2frame.txt & seq2sample.txt

Training & Testing

Training

Finetune the additional link/start-end branches based on a pretrained detection model:

python tools/train.py --data_root ${DATA_ROOT} --ckpt ${PRETRAINED_MODEL} --finetune --batch_size ${BATCH_SIZE} --output_dir ${OUTPUT}

If you want to train with multiple GPUs, add the --mgpus option.
If you want to jointly train the detection and correlation models, remove the --finetune option.

Testing

Evaluate the tracking performance on the validation set:

python tools/eval.py --data_root ${DATA_ROOT} --det_output ${DETECTION_OUTPUT} --ckpt ${CKPT}

Visualization

Please try the code under tools/visualization directory to visualize your 3D object tracking results and make an impressive video!

License

JMODT is released under the MIT license.

Acknowledgement

The object detection module of JMODT is based on EPNet and OpenPCDet. The data association module is based on mmMOT. Many thanks to their official implementation.

Citation

TODO

jmodt's People

Contributors

Stargazers

Watchers

jmodt's Issues

The problem of finetune the model jmodt.pth

I think the optimizer.pth of model jomdt.pth is needed to finetune it.

val_loss_epoch

Traceback (most recent call last):
File "tools/train.py", line 164, in
main()
File "tools/train.py", line 157, in main
val_loader
File "/home/my_com/virtualenv/JMODT/jmodt/utils/train_utils.py", line 198, in train
prev_val_loss = val_loss_epoch

How can i solve it?

Feat Visualization Issues

could you please tell me how to visuallize the feat(.npy), I have try to save as png, but I can not see anything...

UnboundLocalError: local variable 'val_loss_epoch' referenced before assignment

I want to train this code for dataset, following as

$ python tools/train.py --data_root /home/my_com/dataset/KITTI/ --batch_size 4

when first epoch is done, I got error

Can I get some solution from this problem?

Thanks.

Quite a small sized images (.png) in the jmodt folder for visualization.

When i try to visualize the jmodt.avi video by running the viewer.py file, i faced this small sized images in the jmodt folder. Same is the case in the video as well. Please see the attached file.

The parameters for Affinity Compution and data_association.

@Kemo-Huang @P16101150
Excuse me, sir， sorry to bother you. I have several problems with the code with JMODT.
1st.
In the paper the affinity computation part：
Equation(7) the refined affinity X^aff = αA^app + βA^diou

α+β=1； I can not found that the value of α and β. But In the Experiment Results part, you set β=10α for affinity computation， I am confused that which parameter means α or β.
2nd.
In the Experiment Results part:
w^aff = 22, that I can not find which parameter is w^aff, and in data_association I did not find too.

3rd.
In algorithm 1:
where is the X^aff ← αA^app + βA^diou in the program？

I checked data_association.py, does " link_matrix = link_score * w_app + iou_matrix * w_iou + dis_matrix * w_dis" is X^aff ? If it is yes，which is α and which is β，and α+β ≠1.
The last:
if " link_matrix = link_score * w_app + iou_matrix * w_iou + dis_matrix * w_dis" is X^aff, does the X^aff is only used in evaluation step？ Or does it means that X^aff is not used in the train setp， that the training step only used the correlation feature, or only used the A^app？

Thanks, a lot.

How to generate the three files test.txt, sample2frame.txt and seq2sample.txt?

By modifying kitti_converter.py, I can only generate the three files in the picture below. Excuse me, test.txt, sample2frame.txt and seq2sample.txt, how to generate these three files? thank you very much！！

Link to the paper

Hello, the link to the paper no longer exists, can you provide it again?

how to use the dataset

Thanks for your work, recently, I wanted to train the tracking model, so I downloaded the dataset, I can't find the TRACK_OBJECT folder, maybe you could teach me where is the folder I could download. Thank you in advance.

loss is zero

when i start training,why the rcnn_loss is zero?

Hello, where can I find these txt files? Or how do I need to create it? thanks！！

error: protobuf 3.19.6 is installed but protobuf>=4.21.5 is required by {'ortools'}

I am trying to Build and install the required CUDA modules via PyTorch and the CUDA toolkit: sudo python3 setup.py develop , but encountering the following errors: Installed /home/adeel/JMODT
Processing dependencies for jmodt==1.0.0+82f36cd
error: protobuf 3.19.6 is installed but protobuf>=4.21.5 is required by {'ortools'}

‘best_model.path’

Thank you very much for your work. I noticed that the best checkpoint doesn't seem to have been updated while I was training, is this normal?

About the Train Seq and Val Seq.

@Kemo-Huang

sorry to bother you.
In the paper, it shows that The training sequences are split into a training set and a validation set with roughly equal number of frames. Specifically, the training set has 10 sequences and 3975 frames, and the validation set contains 11 sequences and 3945 frames.
But the code split the dataset into a training set with 10 sequences that contain 3995 frames， and the validation set with 10 sequences that have 3864 frames.
If the seventeenth sequence were added to the validation set, it would contain 4009 frames.

So may I have the sequences that were used in your Paper， please？
If you delete some frames when you train or validate please tell me，Thanks a lot.

I check each sequence in the Kitti dataset. Got the number of each Sequence like this.

Seq-ID	number of frames
0	154
1	443
2	233
3	144
4	314
5	297
6	270
7	800
8	390
9	803
10	294
11	373
12	78
13	340
14	106
15	376
16	209
17	145
18	339
19	1059
20	837

Multiple GPUs for training

Hello, when I used multiple GPUs for training, it reported the following error, how should I solve it?