** still a work in progress, we will upload the training code in the following weeks **
==================================================== This is an initial commit implementing InterpoNet, A brain inspired neural network for optical flow dense interpolation by Shay Zweig and Lior Wolf from Tel Aviv University (link)
InterpoNet achieved state-of-the-art results in November 2016 on the MPI-Sintel and KITTI2012 Optical Flow benchmarks.
The code was developed on Ubuntu 14.04, using Tensorflow. You can see the performance it achieved on the KITTI2012 and MPI-Sintel optical flow scoreboards.
For now only the code for inference is uploaded, we will upload the training code soon.
- Create a python (2.7) virtualenv, by typing:
virtualenv --no-site-packages env
- activate the virtualenv, by typing
source env/bin/activate
- Clone this repository by typing:
git clone https://github.com/shayzweig/InterpoNet
- Install all following python packages (using
pip install
) : numpy (tested version: 1.11.2), scikit-image (tested version: 0.12.3). (other dependencies such as cython might be required) - install tensorflow according to the instructions in their website . We only tested our program using tensorflow 0.10.0, and 0.11.0, it should work for versions >0.10.0
- Make sure to configure tensorflow to your needs (GPU usage preferred, you will need specific versions of cuda and cudnn for that)
- Install the variational inference program:
- Libraries libpng, libm and liblapack are required.
- from the root folder of Interponet, type:
cd SrcVariational
- type:
make
- the comilation should run without errors.
Test your installation by running the following command from the root folder of InterpoNet:
python InterpoNet.py example/frame_0001.png example/frame_0002.png example/frame_0001.dat example/frame_0001.txt example/frame_0001.flo --ba_matches_filename=example/frame_0001_BA.txt --sintel
No errors should be displayed
The program was only tested under a 64-bit Linux distribution with tensorflow version 0.10 . We do not give any support for compilation issues or other OS.
The input to the algorithm is divided into three components:
-
The two images in the image pair they must have the same shape and they are only used for the variational energy minimization.
-
A matching file produced by a matching algorithm. The format of the matching file should be as follows: A text file, each match should be preseted in a different row which should include the source and target coordinates seperated with spaces: x1 y1 x2 y2
Any additional information in the row (such as match score) is discarded.
To compute the matches - you can use any matching algorithm. The best results we obtained were on [FlowFields] (https://www.dfki.de/web/research/publications?pubid=7987)
- Edges file produced by SED. The edges should be calculated using the SED algorithm. You can download the code from these links.
- http://research.microsoft.com/en-us/downloads/389109f6-b4e8-404c-84bf-239f7cbf4e3d/
- that will require also Piotr Dollar's toolbox: http://vision.ucsd.edu/~pdollar/toolbox/doc/index.html
To run the InterpoNet pipeline, use the following syntax:
python interponet.py <img1_filename> <img2_filename> <edges_filename> <matches_filename> <out_filename> [optional --model_filename model_filename] [optional --ba_matches_filename ba_matches_filename] [optional --img_width img_width] [optional --img_height img_height] [optional --downscale downscale] [optional --sintel]
Example use:
python InterpoNet.py example/frame_0001.png example/frame_0002.png example/frame_0001.dat example/frame_0001.txt example/frame_0001.flo --ba_matches_filename=example/frame_0001_BA.txt --sintel
Mandatory:
- img1_filename : The first image in the image pair - used in the variational post processing
- img2_filename : The second image in the image pair - used in the variational post processing
- edges_filename : The edges extracted by the SED edge detector
- matches_filename : The output of the matching algorithm given in the format:
- out_filename : The output filename
optional:
- model_filename [default='models/ff_sintel.ckpt'] : The filename of the model to use.
- ba_matches_filename [default=None] : Matching file from second image in the pair to the first (B->A)
- img_width [default=1024] : The width of the flow map.
- img_height [default=436] : The height of the flow map.
- downscale [default=8] : How much o downscale the image before running the model - 8 is the recommended value.
- sintel : Use default parameters for the sintel dataset: model_name='models/ff_sintel.ckpt', img_width=1024, img_height=436, downscale=8.
The output flow file is in the same format as the flow maps in the MPI-Sintel dataset. The provided function: load_flow_file in io_utils.py loads the file into an h x w x 2 matrix.
- ff_sintel - Pretrained on flowfields flying chairs and finetuned on flowfields sintel.
- df_fitti2012 - Pretrained on flowfields flying chairs and finetuned on discrete flow KITTI2012.
InterpoNet was built with the help of the following great software pieces:
- TensorFlow
- [EpicFlow] (https://thoth.inrialpes.fr/src/epicflow/)