This is the official Pytorch implementation of the TPAMI 2024 paper Dense Continuous-Time Optical Flow from Event Cameras.
If you find this code useful, please cite us:
@Article{Gehrig2024pami,
author = {Mathias Gehrig and Manasi Muglikar and Davide Scaramuzza},
title = {Dense Continuous-Time Optical Flow from Event Cameras},
journal = {{IEEE} Trans. Pattern Anal. Mach. Intell. (T-PAMI)},
year = 2024
}We highly recommend to use Mambaforge to reduce the installation time.
conda create -y -n bflow python=3.11 pip
conda activate bflow
conda config --set channel_priority flexible
CUDA_VERSION=12.1
conda install -y h5py=3.10.0 blosc-hdf5-plugin=1.0.0 llvm-openmp=15.0.7 \
hydra-core=1.3.2 einops=0.7 tqdm numba \
pytorch=2.1.2 torchvision pytorch-cuda=$CUDA_VERSION \
-c pytorch -c nvidia -c conda-forge
python -m pip install pytorch-lightning==2.1.3 wandb==0.16.1 \
opencv-python==4.8.1.78 imageio==2.33.1 lpips==0.1.4 \
pandas==2.1.4 plotly==5.18.0 moviepy==1.0.3 tabulate==0.9.0 \
loguru==0.7.2 matplotlib==3.8.2 scikit-image==0.22.0 kaleido| Train | Val | |
|---|---|---|
| pre-processed dataset | download | download |
| Train | Test (input) | |
|---|---|---|
| pre-processed dataset | download | download |
| crc32 | c1b618fc | ffbacb7e |
| Events only | Events + Images | |
|---|---|---|
| pre-trained checkpoint | download | download |
| md5 | 61e102 | 2ce3aa |
| Events only | Events + Images | |
|---|---|---|
| pre-trained checkpoint | download | download |
| md5 | d17002 | 05770b |
- Set
DATA_DIRas the path to the MultiFlow dataset (parent of train and val dir) - Set
MDL_CFG=E_I_LU5_BD10_lowpyramidto use both events and frames, orMDL_CFG=E_LU5_BD10_lowpyramidto use only events
- Set
LOG_ONLY_NUMBERS=trueto avoid logging images (can require a lot of space). Set to false by default.
GPU_ID=0
python train.py model=raft-spline dataset=multiflow_regen dataset.path=${DATA_DIR} wandb.group_name=multiflow \
hardware.gpus=${GPU_ID} hardware.num_workers=6 +experiment/multiflow/raft_spline=${MLD_CFG} \
logging.only_numbers=${LOG_ONLY_NUMBERS}- Set
DATA_DIRas the path to the DSEC dataset (parent of train and test dir) - Set
MDL_CFG=E_I_LU4_BD2_lowpyramidto use both events and frames, orMDL_CFG=E_LU4_BD2_lowpyramidto use only events
- Set
LOG_ONLY_NUMBERS=trueto avoid logging images (can require a lot of space). Set to false by default.
GPU_ID=0
python train.py model=raft-spline dataset=dsec dataset.path=${DATA_DIR} wandb.group_name=dsec \
hardware.gpus=${GPU_ID} hardware.num_workers=6 +experiment/dsec/raft_spline=${MLD_CFG} \
logging.only_numbers=${LOG_ONLY_NUMBERS}- Set
DATA_DIRas the path to the MultiFlow dataset (parent of train and val dir) - Set
MDL_CFG=E_I_LU5_BD10_lowpyramidto use both events and frames, orMDL_CFG=E_LU5_BD10_lowpyramidto use only events
- Set
CKPTto the path of the correct checkpoint
GPU_ID=0
python val.py model=raft-spline dataset=multiflow_regen dataset.path=${DATA_DIR} hardware.gpus=${GPU_ID} \
+experiment/multiflow/raft_spline=${MLD_CFG} checkpoint=${CKPT}work in progress
This project has used code from RAFT for parts of the model architecture.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent