maet3608 / nuts-ml Goto Github PK

Flow-based data pre-processing for deep learning

License: Other

Python 96.99% Shell 0.30% Makefile 2.72%

deep-learning deep-learning-framework deep-learning-library data-science data-preprocessing

nuts-ml's Introduction

nuts-ml is a data pre-processing library for GPU based deep learning that provides common pre-processing functions as independent, reusable units. These so called 'nuts' can be freely arranged to build data flows that are efficient, easy to read and modify.

The following example gives a taste of a nuts-ml data-flow that trains a network on image data and prints training loss and accuracy

(train_samples >> Stratify(1) >> read_image >> transform >> augment >> 
   Shuffle(100) >> build_batch >> network.train() >>  
   Print('train loss:{} acc:{}') >> Consume())

nuts-ml is based on nuts-flow, which is described here.

nuts-ml's People

Stargazers

Watchers

Forkers

codeaudit dsuarezv ruwant allensmile benjamesbabala hunslater-deeplearning xinering reddimohan prashanthmuthurajaiah bahmantahayori

nuts-ml's Issues

Enable test for PlotLines

Test for PlotLines in tests/nutsml/plotter.py is currently disabled because the test runs fine individually but fails when running as part of the test suite. Unclear if that is a matplotlib or pytest issue.

Cifar example crashes with tensorflow backend

File "", line 1, in
runfile('/Users/kerryhalupka/nuts-ml/nutsml/examples/cifar/cnn_train.py', wdir='/Users/kerryhalupka/nuts-ml/nutsml/examples/cifar')

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile
execfile(filename, namespace)

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile
builtins.execfile(filename, *where)

File "/Users/kerryhalupka/nuts-ml/nutsml/examples/cifar/cnn_train.py", line 122, in
train()

File "/Users/kerryhalupka/nuts-ml/nutsml/examples/cifar/cnn_train.py", line 117, in train
network.evaluate([categorical_accuracy]))

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/nutsflow/factory.py", line 43, in rrshift
return func(*args, **self.kwargs)

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/nutsml/network.py", line 100, in EvalNut
results = tuple(compute_metric(m, targets, preds) for m in metrics)

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/nutsml/network.py", line 100, in
results = tuple(compute_metric(m, targets, preds) for m in metrics)

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/nutsml/network.py", line 85, in compute_metric
result = metric(targets, preds)

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/keras/metrics.py", line 25, in categorical_accuracy
return K.cast(K.equal(K.argmax(y_true, axis=-1),

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1339, in argmax
axis = _normalize_axis(axis, ndim(x))

File "/Users/kerryhalupka/anaconda/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 516, in ndim
dims = x.get_shape()._dims

AttributeError: 'numpy.ndarray' object has no attribute 'get_shape'

Add wrapper for TFLearn

Add TFLearnNetwork(model) to wrap tensorflow models created with TFLearn: http://tflearn.org/

Improved error messages

Due to the chained, wrapped iterators wrongly composed pipelines often produce cryptic error messages. Adding assertions on input data and catching common errors should lead to more informative messages.

Badly formatted Sphinx documentation

Some docstrings lead to badly formatted Sphinx documentation because newlines in docstrings are not preserved in Sphinx HTML

Faster conversion from PIL image to numpy array

Current implementation in nutsml.imageutil.pil_to_arr(image) could be improved. See
https://blog.eduardovalle.com/2015/08/25/input-images-theano/

ShowImage does not show image when prefetching is active

When building a pipeline such as ShowImage() >> BuildBatch() no images are displayed because BuildBatch performs prefetching. This is due to a threading problem with matplotlib, which is used for image display.

Workaround: Set prefetching to 0 for bathc builder: BuildBatch(prefetch=0)

Coverage badge shows 70% but should show 100%

see README.rst

Add wrapper for Pytorch

Implement NetworkPytorch wrapper

ViewImage and ViewImageAnnotation do not draw when followed by Prefetch/BuildBatch

for instance, this does not work (but should)
load_images >> ViewImage() >> Prefetch() >> Consume()

The following works:
load_images >> Prefetch() >> ViewImage() >> Consume

Similarly the moment a BuildBatch() appears somewhere after ViewImage()
drawing does not work because BuildBatch internally uses the same code as Prefetch().
This is threading issue due to the fact that matplotlib is not thread safe.

Add other/improved method to balance class distributions

https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29

Remove deprecated BuildBatch.by()

BuildBatch.by() has been replaced by BuildBatch.input() and BuildBatch.output()

Get output/activation of a network layer

Example code for Keras:

@nut_function
def GetActivations(batch, model, layer, istrain=1):
from keras import backend as K

f = K.function([model.input, K.learning_phase()],
               [model.get_layer(layer).output])
return f([batch[0], istrain])

build_batch >> GetActivations(network.model) >> FlattenCol(0) >> Collect()

Having a network function for it would be nice, e.g.
build_batch >> network.activation('layername') >> Collect()

Issue running on OS X 10.11.6 with Anaconda

I had issues installing nuts-ml from source on OS X 10.11.6 using Anaconda.
It ran on properly on an AWS GPU Unbuntu 14.04LTS instance.

Here's the output from OS X.

David-Laxers-MacBook-Pro:~ davidlaxer$ pip install nutsml
Collecting nutsml
  Downloading nutsml-1.0.19-py2-none-any.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 274kB/s 
Requirement already satisfied: pandas>=0.18.1 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Requirement already satisfied: scipy>=0.17.0 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Requirement already satisfied: matplotlib>=1.5.1 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Collecting pyyaml>=3.12 (from nutsml)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |████████████████████████████████| 256kB 659kB/s 
Collecting xlrd>=1.0.0 (from nutsml)
  Downloading xlrd-1.0.0.tar.gz (2.6MB)
    100% |████████████████████████████████| 2.6MB 149kB/s 
Collecting dplython>=0.0.7 (from nutsml)
  Downloading dplython-0.0.7.tar.gz (715kB)
    100% |████████████████████████████████| 716kB 437kB/s 
Requirement already satisfied: six>=1.10.0 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Collecting nutsflow>=1.0.14 (from nutsml)
  Downloading nutsflow-1.0.14-py2-none-any.whl
Requirement already satisfied: pillow>=3.0.0 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Requirement already satisfied: scikit-image>=0.12.3 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Collecting pytest>=3.0.3 (from nutsml)
  Downloading pytest-3.1.1-py2.py3-none-any.whl (179kB)
    100% |████████████████████████████████| 184kB 843kB/s 
Requirement already satisfied: numpy>=1.11.1 in ./anaconda/lib/python2.7/site-packages (from nutsml)
Requirement already satisfied: pytz>=2011k in ./anaconda/lib/python2.7/site-packages (from pandas>=0.18.1->nutsml)
Requirement already satisfied: python-dateutil in ./anaconda/lib/python2.7/site-packages (from pandas>=0.18.1->nutsml)
Requirement already satisfied: cycler>=0.10 in ./anaconda/lib/python2.7/site-packages (from matplotlib>=1.5.1->nutsml)
Requirement already satisfied: pyparsing!=2.0.0,!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in ./anaconda/lib/python2.7/site-packages (from matplotlib>=1.5.1->nutsml)
Requirement already satisfied: subprocess32 in ./anaconda/lib/python2.7/site-packages (from matplotlib>=1.5.1->nutsml)
Requirement already satisfied: functools32 in ./anaconda/lib/python2.7/site-packages (from matplotlib>=1.5.1->nutsml)
Requirement already satisfied: olefile in ./anaconda/lib/python2.7/site-packages (from pillow>=3.0.0->nutsml)
Requirement already satisfied: networkx>=1.8 in ./anaconda/lib/python2.7/site-packages (from scikit-image>=0.12.3->nutsml)
Requirement already satisfied: dask[array]>=0.5.0 in ./anaconda/lib/python2.7/site-packages (from scikit-image>=0.12.3->nutsml)
Collecting py>=1.4.29 (from pytest>=3.0.3->nutsml)
  Downloading py-1.4.33-py2.py3-none-any.whl (83kB)
    100% |████████████████████████████████| 92kB 1.6MB/s 
Requirement already satisfied: setuptools in ./anaconda/lib/python2.7/site-packages (from pytest>=3.0.3->nutsml)
Requirement already satisfied: toolz>=0.7.2 in ./anaconda/lib/python2.7/site-packages (from dask[array]>=0.5.0->scikit-image>=0.12.3->nutsml)
Requirement already satisfied: appdirs>=1.4.0 in ./anaconda/lib/python2.7/site-packages (from setuptools->pytest>=3.0.3->nutsml)
Requirement already satisfied: packaging>=16.8 in ./anaconda/lib/python2.7/site-packages (from setuptools->pytest>=3.0.3->nutsml)
Building wheels for collected packages: pyyaml, xlrd, dplython
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /Users/davidlaxer/Library/Caches/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for xlrd ... done
  Stored in directory: /Users/davidlaxer/Library/Caches/pip/wheels/40/d4/6c/df6603e86ef3183ba2ecc97c5c3f1bf92802d54aa939522235
  Running setup.py bdist_wheel for dplython ... done
  Stored in directory: /Users/davidlaxer/Library/Caches/pip/wheels/dd/50/ba/1ff1d4b5dd4c5939da18da3efcb7d94c70cb9d331e8db773d5
Successfully built pyyaml xlrd dplython
Installing collected packages: pyyaml, xlrd, dplython, py, pytest, nutsflow, nutsml
  Found existing installation: PyYAML 3.11
    DEPRECATION: Uninstalling a distutils installed project (pyyaml) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling PyYAML-3.11:
      Successfully uninstalled PyYAML-3.11
  Found existing installation: xlrd 0.9.3
    DEPRECATION: Uninstalling a distutils installed project (xlrd) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling xlrd-0.9.3:
      Successfully uninstalled xlrd-0.9.3
  Found existing installation: py 1.4.27
    Uninstalling py-1.4.27:
      Successfully uninstalled py-1.4.27
  Found existing installation: pytest 2.7.1
    Uninstalling pytest-2.7.1:
      Successfully uninstalled pytest-2.7.1
Successfully installed dplython-0.0.7 nutsflow-1.0.14 nutsml-1.0.19 py-1.4.33 pytest-3.1.1 pyyaml-3.12 xlrd-1.0.0
David-Laxers-MacBook-Pro:~ davidlaxer$ pwd
/Users/davidlaxer
David-Laxers-MacBook-Pro:~ davidlaxer$ git clone https://github.com/maet3608/nuts-ml.git
Cloning into 'nuts-ml'...
remote: Counting objects: 3313, done.
remote: Compressing objects: 100% (922/922), done.
sutReceiving objects:  31% (1043/3313), 4.82 MiB | 1010.00 KiB/s   
remote: Total 3313 (delta 2244), reused 3313 (delta 2244), pack-reused 0
Receiving objects: 100% (3313/3313), 26.79 MiB | 977.00 KiB/s, done.
Resolving deltas: 100% (2244/2244), done.
David-Laxers-MacBook-Pro:~ davidlaxer$ cd nuts-ml/
David-Laxers-MacBook-Pro:nuts-ml davidlaxer$ python setup.py install
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    import nutsml
  File "/Users/davidlaxer/nuts-ml/nutsml/__init__.py", line 9, in <module>
    from nutsml.plotter import PlotLines
  File "/Users/davidlaxer/nuts-ml/nutsml/plotter.py", line 8, in <module>
    import matplotlib.pyplot as plt
  File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)
  File "/Users/davidlaxer/anaconda/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py", line 19, in <module>
    from matplotlib.backends import _macosx
RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.
David-Laxers-MacBook-Pro:nuts-ml davidlaxer$ python
Python 2.7.11 |Anaconda custom (x86_64)| (default, Dec  6 2015, 18:57:58) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
>>> 
David-Laxers-MacBook-Pro:nuts-ml davidlaxer$ ipython 
Python 2.7.11 |Anaconda custom (x86_64)| (default, Dec  6 2015, 18:57:58) 
Type "copyright", "credits" or "license" for more information.

IPython 5.3.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import nuts-ml
  File "<ipython-input-1-1522a71e241c>", line 1
    import nuts-ml
               ^
SyntaxError: invalid syntax


In [2]: import nutsml
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-851da473b9fb> in <module>()
----> 1 import nutsml

/Users/davidlaxer/nuts-ml/nutsml/__init__.py in <module>()
      7 from nutsml.logger import LogToFile, LogCols
      8 from nutsml.network import Network, KerasNetwork, LasagneNetwork
----> 9 from nutsml.plotter import PlotLines
     10 from nutsml.reader import DplyToList, ReadImage, ReadLabelDirs, ReadPandas
     11 from nutsml.transformer import (TransformImage, AugmentImage, ImageMean,

/Users/davidlaxer/nuts-ml/nutsml/plotter.py in <module>()
      6 
      7 import itertools as itt
----> 8 import matplotlib.pyplot as plt
      9 
     10 from nutsflow import NutFunction

/Users/davidlaxer/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py in <module>()
    113 
    114 from matplotlib.backends import pylab_setup
--> 115 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    116 
    117 _IP_REGISTERED = None

/Users/davidlaxer/anaconda/lib/python2.7/site-packages/matplotlib/backends/__init__.pyc in pylab_setup()
     30     # imports. 0 means only perform absolute imports.
     31     backend_mod = __import__(backend_name,
---> 32                              globals(),locals(),[backend_name],0)
     33 
     34     # Things we pull in from all backends

/Users/davidlaxer/anaconda/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in <module>()
     17 
     18 import matplotlib
---> 19 from matplotlib.backends import _macosx
     20 
     21 from .backend_agg import RendererAgg, FigureCanvasAgg

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

In [3]: quit()
David-Laxers-MacBook-Pro:nuts-ml davidlaxer$ pip install nutsflow
Requirement already satisfied: nutsflow in /Users/davidlaxer/anaconda/lib/python2.7/site-packages
Requirement already satisfied: six>=1.10.0 in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from nutsflow)
Requirement already satisfied: pytest>=3.0.3 in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from nutsflow)
Requirement already satisfied: py>=1.4.29 in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from pytest>=3.0.3->nutsflow)
Requirement already satisfied: setuptools in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from pytest>=3.0.3->nutsflow)
Requirement already satisfied: appdirs>=1.4.0 in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from setuptools->pytest>=3.0.3->nutsflow)
Requirement already satisfied: packaging>=16.8 in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from setuptools->pytest>=3.0.3->nutsflow)
Requirement already satisfied: pyparsing in /Users/davidlaxer/anaconda/lib/python2.7/site-packages (from packaging>=16.8->setuptools->pytest>=3.0.3->nutsflow)

transforming multi-class segmentation ground-truth mask result in label interpolation

The resulting mask image, after transforming a ground-truth mask (for multi-class segmentation) with AugmentImage will contain label values that were not present in the original mask. Looks like it happens due to interpolation.

from nutsflow import *
from nutsml import *
import numpy as np
a = np.zeros((100,100), dtype=np.int8)
a[40:60, 40:60] = 4

augment_2 = (AugmentImage((1,)).by('rotate', 1.0, [10, 50]))
img_viewer = ViewImage(imgcols=(0, 1), layout=(1, 2), pause=10)
[(a, a),] >> augment_2 >> img_viewer >> Consume()

nut decorators hide docstrings of wrapped functions

help(PrintTypeInfo)
class Wrapper(nutsflow.base.NutFunction)
| Method resolution order:
| Wrapper
| nutsflow.base.NutFunction
| nutsflow.base.Nut
| builtin.object
|

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.