The implementations of PanoFlow: Learning Optical Flow for Panoramic Images. We achieve state-of-the-art accuracy on the public OmniFlowNet dataset and the proposed FlowScape (Flow360) dataset. This repository is built on the basis of CSFlow.
From left to right: overlapping image pairs, optical flow, and semantics.
FlowScape (Flow360) dataset consists of 8 various city maps in four weathers:
sunny, fog, cloud, and rain.
We collect 100 consecutive panoramic images at each random position,
resulting in a total of 6,400 frames with a resolution of 1024 x 512 ,
each with optical flow ground truth and semantic labels,
which can be used for training and evaluation. In the current release,
we provide optical flow ground truth in the classic format (i.e. the traditional flow).
If you need 360° flow ground truth, you can simply convert it refer to the
paper.
Since the flow field of panoramic images usually contains large displacement that interferes with visualization and fades colors, we modified the visualization method of optical flow, and lowered the color saturation of optical flow greater than the threshold.
better_flow_to_image(flow, alpha=0.1, max_flow=25)
The function can be found in the flow_utils.py. In our paper, we set the alpha=0.1, max_flow=25.
The valid mask excludes pixels whose semantics are sky.
The semantic labels are as following:
camvid_colors = OrderedDict([
("Unlabeled", np.array([0, 0, 0], dtype=np.uint8)),
("Building", np.array([70, 70, 70], dtype=np.uint8)),
("Fence", np.array([100, 40, 40], dtype=np.uint8)),
("Other", np.array([55, 90, 80], dtype=np.uint8)),
("Pedestrian", np.array([220, 20, 60], dtype=np.uint8)),
("Pole", np.array([153, 153, 153], dtype=np.uint8)),
("RoadLine", np.array([157, 234, 50], dtype=np.uint8)),
("Road", np.array([128, 64, 128], dtype=np.uint8)),
("SideWalk", np.array([244, 35, 232], dtype=np.uint8)),
("Vegetation", np.array([107, 142, 35], dtype=np.uint8)),
("Vehicles", np.array([0, 0, 142], dtype=np.uint8)),
("Wall", np.array([102, 102, 156], dtype=np.uint8)),
("TrafficSign", np.array([220, 220, 0], dtype=np.uint8)),
("Sky", np.array([70, 130, 180], dtype=np.uint8)),
("Ground", np.array([81, 0, 81], dtype=np.uint8)),
("Bridge", np.array([150, 100, 100], dtype=np.uint8)),
("RailTrack", np.array([230, 150, 140], dtype=np.uint8)),
("GroundRail", np.array([180, 165, 180], dtype=np.uint8)),
("TrafficLight", np.array([250, 170, 30], dtype=np.uint8)),
("Static", np.array([110, 190, 160], dtype=np.uint8)),
("Dynamic", np.array([170, 120, 50], dtype=np.uint8)),
("Water", np.array([45, 60, 150], dtype=np.uint8)),
("Terrain", np.array([145, 170, 100], dtype=np.uint8)),
])
Anyone can download our FlowScape (Flow360) dataset via these links.
Download link 1 (Tencent WeiYun):
https://share.weiyun.com/SoXICYgh
Download link 2 (Baidu Cloud):
https://pan.baidu.com/s/1ZjY6J-zN5Wb7JxRMeHvQSw?pwd=7u2v
The content in the above links are consistent, if you encounter network problems, you can try switching to the other link.
python setup.py develop
The pretrained model that the paper used can be found there:
https://share.weiyun.com/SIpeQTNE
Notice that the checkpoints don‘t consist of the CFE, considering that CFE is an estimation method, you only need to turn it on while inferring to obtain the 360° flow.
To train, use the following command format:
python ./tools/train.py
--model PanoFlow(CSFlow)
--dataset Flow360
--data_root $YOUR_DATA_PATH$
--batch_size 6
--name PanoFlow(CSFlow)-test
--validation Chairs
--val_Chairs_root $YOUR_DATA_PATH$
--num_steps 100
--lr 0.000125
--image_size 400 720
--wdecay 0.0001
To eval, use the following command format:
python ./tools/eval.py
--model PanoFlow(CSFlow)
--restore_ckpt ./checkpoints/PanoFlow(CSFlow)-wo-CFE.pth
--CFE
--validation Flow360
--val_Flow360_root $YOUR_DATA_PATH$
For more details, please check the code or refer our paper.
* local: you should create this folder in your local repository and these folders will not upload to remote repository.
├── data (local) # Store test/training data
├── checkpoints (local) # Store the checkpoints
├── runs (local) # Store the training log
├── opticalflow # All source code
| ├─ api # Called by tools
| ├─ core # Core code call by other directorys. Provide dataset, models ....
| | ├─ dataset # I/O of each dataset
| | ├─ model # Models, includeing all the modules that derive nn.module
| | ├─ util # Utility functions
├── tools # Scripts for test and train
├── work_dirs (local) # For developers to save thier own codes and assets
If you find our project helpful in your research, please cite with:
@article{shi2022panoflow,
title={PanoFlow: Learning optical flow for panoramic images},
author={Shi, Hao and Zhou, Yifan and Yang, Kailun and Ye, Yaozu and Yin, Xiaoting and Yin, Zhe and Meng, Shi and Wang, Kaiwei},
journal={arXiv preprint arXiv:2202.13388},
year={2022}
}
Hao Shi，YiFan Zhou
If you have any questions, welcome to e-mail me: [email protected], and I will try my best to help you. =)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent