f4z3r / cil_project Goto Github PK

Resources

This is an example of exactly what we are supposed to do. The technique used can be read in the report within the git repo.

This is another repo using tensorflow to implement the same problem.

This paper does more broad per-pixel segmentation by using satellite images to classify complete regions. However, this is also applicable for road segmentation.

Papers

The BibTex references bellow refer to papers in papers/ directory.

• @article{panboonyuen2017road,
    title={Road Segmentation of Remotely-Sensed Images Using Deep Convolutional Neural Networks with Landscape Metrics and Conditional Random Fields},
    author={Panboonyuen, Teerapong and Jitkajornwanich, Kulsawasd and Lawawirojwong, Siam and Srestasathiern, Panu and Vateekul, Peerapon},
    journal={Remote Sensing},
    volume={9},
    number={7},
    pages={680},
    year={2017},
    publisher={Multidisciplinary Digital Publishing Institute}
  }
  

• @article{bhattacharya1997improved,
    title={An improved backpropagation neural network for detection of road-like features in satellite imagery},
    author={Bhattacharya, U and Parui, SK},
    journal={International Journal of Remote Sensing},
    volume={18},
    number={16},
    pages={3379--3394},
    year={1997},
    publisher={Taylor \& Francis}
  }
  

• @article{mokhtarzade2008automatic,
    title={Automatic road extraction from high resolution satellite images using neural networks, texture analysis, fuzzy clustering   and genetic algorithms},
    author={Mokhtarzade, M and Zoej, MJ Valadan and Ebadi, H},
    journal={The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
    volume={37},
    pages={549--556},
    year={2008}
  }
  

• @article{song2004road,
    title={Road extraction using SVM and image segmentation},
    author={Song, Mingjun and Civco, Daniel},
    journal={Photogrammetric Engineering \& Remote Sensing},
    volume={70},
    number={12},
    pages={1365--1371},
    year={2004},
    publisher={American Society for Photogrammetry and Remote Sensing}
  }
  

• @article{kingma2014adam,
    title={Adam: A method for stochastic optimization},
    author={Kingma, Diederik P and Ba, Jimmy},
    journal={arXiv preprint arXiv:1412.6980},
    year={2014}
  }
  

• @article{xu2015empirical,
    title={Empirical evaluation of rectified activations in convolutional network},
    author={Xu, Bing and Wang, Naiyan and Chen, Tianqi and Li, Mu},
    journal={arXiv preprint arXiv:1505.00853},
    year={2015}
  }
  

• @article{bittel2015pixel,
    title={Pixel-wise segmentation of street with neural networks},
    author={Bittel, Sebastian and Kaiser, Vitali and Teichmann, Marvin and Thoma, Martin},
    journal={arXiv preprint arXiv:1511.00513},
    year={2015}
  }
  

• @article{zhang2006benefit,
    title={Benefit of the angular texture signature for the separation of parking lots and roads on high resolution multi-spectral imagery},
    author={Zhang, Qiaoping and Couloigner, Isabelle},
    journal={Pattern recognition letters},
    volume={27},
    number={9},
    pages={937--946},
    year={2006},
    publisher={Elsevier}
  }
  

• @article{mena2005automatic,
    title={An automatic method for road extraction in rural and semi-urban areas starting from high resolution satellite imagery},
    author={Mena, Juan B and Malpica, Jos{\'e} A},
    journal={Pattern recognition letters},
    volume={26},
    number={9},
    pages={1201--1220},
    year={2005},
    publisher={Elsevier}
  }
  

• @article{henry2018road,
    title={Road Segmentation in SAR Satellite Images with Deep Fully-Convolutional Neural Networks},
    author={Henry, Corentin and Azimi, Seyed Majid and Merkle, Nina},
    journal={arXiv preprint arXiv:1802.01445},
    year={2018}
  }
  

• @article{kaiser2017learning,
    title={Learning Aerial Image Segmentation from Online Maps},
    author={Kaiser, Pascal and Wegner, Jan Dirk and Lucchi, Aur{\'e}lien and Jaggi, Martin and Hofmann, Thomas and Schindler, Konrad},
    journal={IEEE Transactions on Geoscience and Remote Sensing},
    volume={55},
    number={11},
    pages={6054--6068},
    year={2017},
    publisher={IEEE}
  }
  

There are other labelled datasets out there. Find them and adapt them to be used in our implementation.
Example: http://www2.isprs.org/commissions/comm3/wg4/2d-sem-label-potsdam.html

This task involves copying the code of following repository into our own implementation:
https://github.com/mrubash1/keras-semantic-segmentation
They implemented following models:
• FCN (Fully Convolutional Networks) using ResNets
• U-Net
• Fully Convolutional DenseNets (aka the 100 Layer Tiramisu)
Analyse which would be the best for our problem. Then implement it in our implementation.
Alternative: https://github.com/nshaud/DeepNetsForEO

Make project runnable on Windows by using cross-platform pathes:
os.path.normpath("assets/training/data")
os.path.dirname(file_path) instead of '..'

If we train a model several outputs are generated. E.g. logs, model checkpoints, .csv prediction, csv image visualisations, etc.
Because of this the output of each model needs to be directed into one model folder.

When we run:
run.py -train <model_name>

It should create a new folder, e.g.:
cil_project/models/_<model_name>/

This folder and an underlying folder structure should be created. Furthermore, it should be ensured that all outputs are written into the correct folder.

The test set should be predicted and the result needs to be put into a .csv according to the Kaggle competition guidelines.

To start the prediction it could be:
run.py -predict optional:path_to_model.h5 optional:path_to_testset

As input we can provide a model generated by the keras training. This input is optional. If no model is provided, check automatically in the trained models folder, for the last trained/created model and use this.
The input should also optionally provide a parameter to enter the path to the test set (this allows prediction of the validationset). Standard would always be the test folder.

Output: Input each image pad into the predict method of the model (check padding of input image) and predict its label. Save the output in the .csv file according to the Kaggle submission rules. Save the .csv in the model folder provided.

Output : Additionally it would be helpful if the output is also written to an image to have a visual comparison. See issue

See: How can i save a Keras model

Right now each train and validation image is saved in 8 versions.
This results in a high RAM usage and needs a longer time to load for each run. Furthermore it affects further augmentations that we may need in the future.

This is why the EPFL data generator should implement the mirroring and rotating itself.
Randomly select image.
Randomly mirror it around x, y axis.
Randomly rotate it by multiples of 90 degrees.

(I checked this and it does not result in a change of performance. Numpy is so fast in doing this)

In order to be able to add new kinds of models I propose a refactoring of framework.
This includes:
One BaseModel abstract class that does not implement any methods but defines all publicly needed ones (e.g. train, predict, etc.). (TODO: Check if predict should be implemented in BaseModel)

All other models directly extend the BaseModel. There is no more inheritance allowed. This results in more copied code, on the other hand it is more flexible and as the Models will grow we will not need to support older models, etc.

Each model takes two DataGenerators as input. One for the trainset and one for the validationset.
(At the moment this is implemented in the Model class but should be refactored into its own class). These DataGenerators can be fed to Keras.train().

Train the network on Google Colab.

Discuss here any main ideas on the design of the CNN.

Please add ideas specific to the internal design of the CNN to this issue.

Current ideas taken from README

CNN + 4 filters (computation allowing):

• vertical lines (https://www.coursera.org/learn/convolutional-neural-networks/lecture/4Trod/edge-detection-example) -> learn vertical features as streets
• horizontal lines (same above link) -> learn horizontal features as streets
• diagonal lines filter (same above link) -> learn diagonal features as streets
• color (third dimension) -> learn a color threshold given the background possibly (Bayesian estimation)

Objective: learn the features as independent ones. -> threshold on top of the layers mixing all channels.

Convert a submission .csv into an image.

run.py -visualisation <input.csv>

Output: Image in the resolution of the original image that with white colors for roads and black colors for others. Write images into same folder as .csv

This task involves the implementation of the original Full CNN presented in the paper:
“Fully convolutional networks for semantic segmentation” by Long 2015
There are many implementations in all frameworks.
Example: https://github.com/aurora95/Keras-FCN
Implement this model like described in the paper into our implementation. Only support images by 400x400.

Write utility function to perform canny edge detection on images to get more information out of training set.

Enable the augmentation of color saturation, rotation, zooming, contrast, normalization, etc.

Create a function that uses 20% of the test set to create validation set if required.

Improve performance by loading all images into memory before starting the training. This is to overcome the slow current implementation of batching.

See the exact issues in the comments of #7.

There is not a lot of training data available. Therefore we need to enhance our training data. One example is this:
https://github.com/aleju/imgaug

Implement a single run file that handles all executable code.

Desired features:

--augment -g:   augment the training sets (should check that augment was not already run
--train -t:           train the network
--all -a:              augment, train, run
--run -r:             run a trained CNN on the test set and report
--help -h:          display this

Maybe add some config file to store preferences ??

Implement evaluation functions for existing models to check performance on test cases once complete training is performed.

In order that we are all able to run the code online and have a similar development cycle we should test that for everyone the implementation works online.

My suggestion is:

• Installing Google Drive on development machine
• Put whole project folder into Google Drive synchronized folder
• Connect Google Colab with OneDrive

Please test that you can train the model online and that the output is created in the corresponding folder.

If it worked please comment below.

When we execute run.py -train <model_name> a new model is created and Keras trains the model from scratch.
Mostly we do not want to retrain a model from scratch but carry on with the last model.
This is why following functionality should be implemented:
run.py -train-presume optional:path_model.h5

This should loads the model specified by the optional input path to the model. If there is no parameter entered, the implementation should automatically find the last trained model (sorted by date) and load this.
It needs to be ensured that all further output is saved in the corresponding models folder (not a new folder structure should be created. See issue).

To get fast execution speeds on Tensorflow it is the easiest to use precompiled wheel files for your specific CPU/GPU
This is why I would recommend to install one of following wheel files:
https://github.com/mind/wheels

Installation:
https://stackoverflow.com/questions/27885397/how-do-i-install-a-python-package-with-a-whl-file

Implement super naive DNN classifier to benchmark CNN + LR + D solution. See here for implementation info.

Implement a CNN with leaky ReLU activations and Dropout layers after each max pooling.

Sorry mate, but do you mind just using 4 spaces for indentation in python and not use tabs. Mostly because right now cnn_model.py (eg. lines 83-88) has some formatting issues because sometimes you use 4 spaces, but sometimes tabs (which default to 8 characters on Github) so the formatting is out of place.

Implement loaders that load weights from files for already trained networks for all current models.

Maybe find some file naming conventions to ensure that no weights from some model can be loaded into a mismatching model.