Dense Continuous-Time Optical Flow from Event Cameras

This is the official Pytorch implementation of the TPAMI 2024 paper Dense Continuous-Time Optical Flow from Event Cameras.

If you find this code useful, please cite us:

@Article{Gehrig2024pami,
  author        = {Mathias Gehrig and Manasi Muglikar and Davide Scaramuzza},
  title         = {Dense Continuous-Time Optical Flow from Event Cameras},
  journal       = {{IEEE} Trans. Pattern Anal. Mach. Intell. (T-PAMI)},
  year          = 2024
}

Conda Installation

We highly recommend to use Mambaforge to reduce the installation time.

conda create -y -n bflow python=3.11 pip
conda activate bflow
conda config --set channel_priority flexible

CUDA_VERSION=12.1

conda install -y h5py=3.10.0 blosc-hdf5-plugin=1.0.0 llvm-openmp=15.0.7 \
hydra-core=1.3.2 einops=0.7 tqdm numba \
pytorch=2.1.2 torchvision pytorch-cuda=$CUDA_VERSION \
-c pytorch -c nvidia -c conda-forge

python -m pip install pytorch-lightning==2.1.3 wandb==0.16.1 \
opencv-python==4.8.1.78 imageio==2.33.1 lpips==0.1.4 \
pandas==2.1.4 plotly==5.18.0 moviepy==1.0.3 tabulate==0.9.0 \
loguru==0.7.2 matplotlib==3.8.2 scikit-image==0.22.0 kaleido

Data

MultiFlow

	Train	Val
pre-processed dataset	download	download

DSEC

	Train	Test (input)
pre-processed dataset	download	download
crc32	`c1b618fc`	`ffbacb7e`

Checkpoints

MultiFlow

	Events only	Events + Images
pre-trained checkpoint	download	download
md5	`61e102`	`2ce3aa`

DSEC

	Events only	Events + Images
pre-trained checkpoint	download	download
md5	`d17002`	`05770b`

Training

MultiFlow

Set DATA_DIR as the path to the MultiFlow dataset (parent of train and val dir)
Set
- MDL_CFG=E_I_LU5_BD10_lowpyramid to use both events and frames, or
- MDL_CFG=E_LU5_BD10_lowpyramid to use only events
Set LOG_ONLY_NUMBERS=true to avoid logging images (can require a lot of space). Set to false by default.

GPU_ID=0
python train.py model=raft-spline dataset=multiflow_regen dataset.path=${DATA_DIR} wandb.group_name=multiflow \
hardware.gpus=${GPU_ID} hardware.num_workers=6 +experiment/multiflow/raft_spline=${MLD_CFG} \
logging.only_numbers=${LOG_ONLY_NUMBERS}

DSEC

Set DATA_DIR as the path to the DSEC dataset (parent of train and test dir)
Set
- MDL_CFG=E_I_LU4_BD2_lowpyramid to use both events and frames, or
- MDL_CFG=E_LU4_BD2_lowpyramid to use only events
Set LOG_ONLY_NUMBERS=true to avoid logging images (can require a lot of space). Set to false by default.

GPU_ID=0
python train.py model=raft-spline dataset=dsec dataset.path=${DATA_DIR} wandb.group_name=dsec \
hardware.gpus=${GPU_ID} hardware.num_workers=6 +experiment/dsec/raft_spline=${MLD_CFG} \
logging.only_numbers=${LOG_ONLY_NUMBERS}

Evaluation

MultiFlow

Set DATA_DIR as the path to the MultiFlow dataset (parent of train and val dir)
Set
- MDL_CFG=E_I_LU5_BD10_lowpyramid to use both events and frames, or
- MDL_CFG=E_LU5_BD10_lowpyramid to use only events
Set CKPT to the path of the correct checkpoint

GPU_ID=0
python val.py model=raft-spline dataset=multiflow_regen dataset.path=${DATA_DIR} hardware.gpus=${GPU_ID} \
+experiment/multiflow/raft_spline=${MLD_CFG} checkpoint=${CKPT}

DSEC

work in progress

Code Acknowledgments

This project has used code from RAFT for parts of the model architecture.

forexwiki / bflow Goto Github PK

bflow's Introduction

Dense Continuous-Time Optical Flow from Event Cameras

Conda Installation

Data

MultiFlow

DSEC

Checkpoints

MultiFlow

DSEC

Training

MultiFlow

DSEC

Evaluation

MultiFlow

DSEC

Code Acknowledgments

bflow's People

Contributors

Recommend Projects

Recommend Topics

Recommend Org