In current implementation, results of following operations are (possibly broadcasted) element addition :

• Variable + scalar
• scalar + Variable
• Variable + numpy.ndarray (as long as Variable.data + numpy.ndarray is valid)

It is natural to suppose numpy.ndarray + Variable works in the same way.
But return value of numpy.ndarray + Variable is numpy.ndarray whose elements are Variable as shown in the following example.

In [75]: aa = Variable(np.random.uniform(0,1,3))
In [76]: b = np.random.uniform(0,1,3)
In [77]: b + aa
Out[77]: 
array([<chainer.variable.Variable object at 0x7f2a5e1384d0>,
       <chainer.variable.Variable object at 0x7f2a5e138710>,
       <chainer.variable.Variable object at 0x7f2a5e138950>], dtype=object)
In [78]: (b + aa)[0].data
Out[78]: array([ 0.90954313,  0.60971617,  1.05675581])

I think it would be better (b+aa).data equals to b + aa.data

Deep learning framework must have enough performance for fast research and development. In order to decide which framework to use, benchmarks are useful to compare different frameworks before solving heavy learning tasks by oneself.

It is difficult for current Chainer to construct an architecture whose weights are tied (e.g. autoencoder)

ubgpu@ubgpu:~/github/chainer$ python examples/mnist/train_mnist.py
Traceback (most recent call last):
File "examples/mnist/train_mnist.py", line 10, in
from sklearn.datasets import fetch_mldata
File "/usr/local/lib/python2.7/dist-packages/sklearn/datasets/init.py", line 23, in
from .twenty_newsgroups import fetch_20newsgroups
File "/usr/local/lib/python2.7/dist-packages/sklearn/datasets/twenty_newsgroups.py", line 53, in
from ..feature_extraction.text import CountVectorizer
File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/init.py", line 10, in
from . import text
File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", line 29, in
from ..preprocessing import normalize
File "/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/init.py", line 24, in
from .imputation import Imputer
File "/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/imputation.py", line 9, in
from scipy import stats
File "/usr/lib/python2.7/dist-packages/scipy/stats/init.py", line 324, in
from .stats import *
File "/usr/lib/python2.7/dist-packages/scipy/stats/stats.py", line 247, in
from . import distributions
File "/usr/lib/python2.7/dist-packages/scipy/stats/distributions.py", line 17, in
from scipy import optimize
File "/usr/lib/python2.7/dist-packages/scipy/optimize/init.py", line 148, in
from ._root import *
File "/usr/lib/python2.7/dist-packages/scipy/optimize/_root.py", line 20, in
from . import nonlin
File "/usr/lib/python2.7/dist-packages/scipy/optimize/nonlin.py", line 122, in
import scipy.sparse.linalg
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/init.py", line 108, in
from .isolve import *
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/isolve/init.py", line 6, in
from .iterative import *
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/isolve/iterative.py", line 11, in
from scipy.lib.decorator import decorator
ImportError: No module named decorator

                          ^

SyntaxError: invalid syntax
ubgpu@ubgpu:/github/chainer$ sudo pip install scipy
[sudo] password for ubgpu:
You are using pip version 7.0.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): scipy in /usr/lib/python2.7/dist-packages
ubgpu@ubgpu:/github/chainer$

Installing CUDA environment is a difficult task. Users want to know how to recover when an error occurs.

chainer-cuda-requirements command imports cuda.requirements. So, when import chainer fails on invalid environment, the command also fails.
This command needs to work on such a case because a user want to fix his invalid environment.

My idea is to write the requirements text in the script directly.

Travis CI does not support CUDA. We need to use Jenkins.

Suggested by @chrisemoody at twitter. It is desirable to implement GRU unit to cover various kind of architecture.

(Model Zoo)[https://github.com/BVLC/caffe/wiki/Model-Zoo] is a great place to use existing researches for further researches and applications. It will enhance the usability.

The interface should be generalized to models added into Model Zoo in the future, though we do not need perfect compatibility to Caffe (especially on some functionalities of Caffe not used by the models in Model Zoo).

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for gradient_check, especially,

• gradient_check.numerical_grad

Core functionalities like Variable, Function, FunctionSet, and Optimizer are not tested directly.

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for Optimizer, especially,

• Optimizer.setup
• Optimizer.init_state
• Optimizer.zero_grads
• Optimizer.compute_grads_norm
• Optimizer.clip_grads
• Optimizer.weight_decay
• Optimizer.accumulate_grads
• Optimizer.update

I tried to use GPU in AWS and failed.
I used g2.2xlarge with Amazon Linux AMI with NVIDIA GRID GPU Driver, which I think the easiest way to use cuda.

The commands I tried:

install Anaconda

wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.2.0-Linux-x86_64.sh
bash Anaconda-2.2.0-Linux-x86_64.sh -b
echo 'export PATH=$HOME/anaconda/bin:$PATH' >> .bashrc
echo 'export PYTHONPATH=$HOME/anaconda/lib/python2.7/site-packages:$PYTHONPATH' >> .bashrc
source .bashrc
conda update conda -y

install chainer

pip install chainer
sudo apt-get install git
git clone https://github.com/pfnet/chainer.git
pip install -r chainer/chainer/requirements/cuda-requirements.txt
python chainer/examples/mnist/train_mnist.py --gpu 1

The error I encountered:

Traceback (most recent call last):
File "chainer/examples/mnist/train_mnist.py", line 40, in
cuda.init(args.gpu)
File "/home/ec2-user/anaconda/lib/python2.7/site-packages/chainer/cuda.py", line 119, in init
device = Device(device)
pycuda._driver.LogicError: cuDeviceGet failed: invalid device

Could you help me?
I think it is better to release the official chainer AMI for GPU instances.

In previous implementation, examples/imagenet/inception.py contains not only GoogLeNet but also inception module, which is building block of GoogLeNet. Now that inception is moved to chainer/functions/, I think we should rename this file (like googlenet.py).

chainer has a ring log but we want a cute mascot.
For example, Hive has so cute mascot: https://hive.apache.org/
Fluentd too: http://www.fluentd.org/

Cute mascot is very important for spreading softwares.

ubgpu@ubgpu:~/github/chainer$ python examples/sentiment/train_sentiment.py --gpu=0
Epoch: 0
loss: 270866.15
11.64 iters/sec, 734.28 sec

Epoch: 1
loss: 232619.31
10.74 iters/sec, 795.38 sec

Epoch: 2
loss: 218530.70
10.43 iters/sec, 819.13 sec

Epoch: 3
loss: 208379.96
10.66 iters/sec, 801.55 sec

Epoch: 4
loss: 199109.27
10.76 iters/sec, 793.75 sec

Train data evaluation:
Traceback (most recent call last):
File "examples/sentiment/train_sentiment.py", line 185, in
evaluate(train_trees)
File "examples/sentiment/train_sentiment.py", line 124, in evaluate
traverse(tree, train=False, evaluate=result)
File "examples/sentiment/train_sentiment.py", line 93, in traverse
left_loss, left = traverse(left_node, train=train, evaluate=evaluate, root=False)
File "examples/sentiment/train_sentiment.py", line 93, in traverse
left_loss, left = traverse(left_node, train=train, evaluate=evaluate, root=False)
File "examples/sentiment/train_sentiment.py", line 93, in traverse
left_loss, left = traverse(left_node, train=train, evaluate=evaluate, root=False)
File "examples/sentiment/train_sentiment.py", line 93, in traverse
left_loss, left = traverse(left_node, train=train, evaluate=evaluate, root=False)
File "examples/sentiment/train_sentiment.py", line 108, in traverse
predict = y.data.argmax(1)
AttributeError: 'GPUArray' object has no attribute 'argmax'
ubgpu@ubgpu:~/github/chainer$

Use Travis-CI to test pull requests.
Note that Travis-CI doesn't support GPUs. So, we can only check CPU tests, and code styles.

related to #4 .

Verifying the graph topology is hard for complex networks, so it is worthwile to generate the visual computational graph based on the actual forward implementation.

I want to run examples in CI environment to check their correctness.

Related #80

To reduce undesired failure of numerical tests like optimizers or backward_{cpu,gnu}, we should add some mechanism in which we run targeted test multiple times and assert fails only if targeted one fails at all attempts.

One of possible implementation is to use decorator like:

@test_n_times(10, 8):
def test_something(self):
    ....

It should include following topics:

• Release policy (revision/minor scheduling and milestones)
• Issue labels
• Link to style guides
• Testing guide
  Any other topics that should be included?

It is sometimes not exactly 1.0. Use assertAlmostEqual.

https://github.com/pfnet/chainer/blob/master/tests/functions_tests/test_hierarchical_softmax.py#L37

Checking the signature (and types of return values) of each function automatically makes it easy to debug type errors. There are three direction of design:

1. Simply add asserts to forward functions.
2. Use decorators.
3. Pass the type information to Function.__init__

It must follow DRY rule and should have flexibility to support "variadic" functions (such as concat).

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for Variable, especially,

• Variable.set_creator
• Variable.unchain_backward.

Example:

In [89]: a = Variable(np.random.uniform(-1, 1, 3))
In [90]: b = Variable(np.random.uniform(-1, 1, 3))
In [91]: c = a.__rsub__(b)
In [92]: c.data
Out[92]: <chainer.variable.Variable at 0x7f2a5e12fa50>
In [93]: c.data.data
Out[93]: array([-0.91797078,  0.28034073,  0.441827  ])

Note that this method is not used as long as Variable.__sub__ is defined.
So it is not problematic if we do not call it directly.

Same is true for __rmul__, __rdiv__, __rpow__

ubgpu@ubgpu:~/github/chainer$ python examples/mnist/train_mnist.py
fetch MNIST dataset
epoch 1
train mean loss=0.27774727401, accuracy=0.914516667798
test mean loss=0.11268276395, accuracy=0.966000006199
epoch 2
train mean loss=0.136812622429, accuracy=0.958366669913
test mean loss=0.0904840060783, accuracy=0.973000006676
epoch 3
train mean loss=0.108749408927, accuracy=0.966883340677
test mean loss=0.073886228106, accuracy=0.976000006199
epoch 4
train mean loss=0.093579410448, accuracy=0.971016674836
test mean loss=0.0809730870945, accur

it is fine!!! but...

ubgpu@ubgpu:/github/chainer$ python examples/mnist/train_mnist.py --gpu=0
fetch MNIST dataset
Traceback (most recent call last):
File "examples/mnist/train_mnist.py", line 40, in
cuda.init(args.gpu)
File "/usr/local/lib/python2.7/dist-packages/chainer/cuda.py", line 107, in init
'The original import error said: ' + str(_import_error))
RuntimeError: CUDA environment is not correctly set up. The original import error said: cannot import name intern
ubgpu@ubgpu:/github/chainer$ which nvcc
/usr/local/cuda/bin/nvcc
ubgpu@ubgpu:/github/chainer$ sudo pip install pyCUDA
[sudo] password for ubgpu:
You are using pip version 7.0.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): pyCUDA in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pytools>=2011.2 in /usr/local/lib/python2.7/dist-packages (from pyCUDA)
Requirement already satisfied (use --upgrade to upgrade): pytest>=2 in /usr/local/lib/python2.7/dist-packages (from pyCUDA)
Requirement already satisfied (use --upgrade to upgrade): decorator>=3.2.0 in /usr/local/lib/python2.7/dist-packages (from pyCUDA)
Requirement already satisfied (use --upgrade to upgrade): appdirs>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from pytools>=2011.2->pyCUDA)
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/dist-packages (from pytools>=2011.2->pyCUDA)
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.25 in /usr/local/lib/python2.7/dist-packages (from pytest>=2->pyCUDA)
ubgpu@ubgpu:/github/chainer$

I attempted to install chainer on Python 3.4, but it failed due to incompatibility with Python 3. The entire Python scientific stack is Python 3 compatible these days, so I'm hoping there are plans to make chainer compatible as well. I'm looking forward to using it.

It is analogous to numpy.split. Since we have already used the name Split for graph manipulation, we should think another name or rename the original Split function.

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for FunctionSet, especially,

• FunctionSet.to_gpu
• FunctionSet.to_cpu

Current MNIST example uses scikit-learn to download MNIST. It is overkill and adds an installation overhead to users and example testing.

The test codes of CPU forward/backward and GPU forward/backward are currently mixed and cannot be executed separately. This makes it hard to verify installation on non-GPU environment.

sum function reduces all elements, while there are many cases that I want to take a summation over some axes. This is analogous to numpy.sum with the axis argument.

The implementation should contain a general function for GPU to compute a reduction over specified axes.

Thanks for sharing the great package. It's nice if we can monitor the real-time progress like othe DL packages, though I only know some of them.

• http://daemonmaker.blogspot.ca/2014/12/monitoring-experiments-in-pylearn2.html
• http://blocks.readthedocs.org/en/latest/plotting.html

I've created an example using Bokeh in sinhrks/chainer@fbc933f :

1. Install Bokeh

   pip install bokeh

2. Run bokeh-server

   $ bokeh-server&

3. Run examples/mnist/train_mnist_monitoring.py

4. Web-browser is automatically open displaying the real-time progress of the training.

   

   Bokeh allows us to perform zoom/move charts interactively.

   

If this can be a focus of the project, would like to discuss how this can be handled (standardize or put it to example / document, etc).

We cannot install from pypi. Rename it!

It should handle inner-product, matrix-vector product, and matrix-matrix product functions. It is better if it supports a batch mode.

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for utils.

PEP8 is one of the most famous coding standard.
https://www.python.org/dev/peps/pep-0008/
And, we can also use style checker.
https://pypi.python.org/pypi/pep8

Current ImageNet example uses the cv2 package to just load JPEG image files. This is overkill.

According to PEP396, the following will be desirable

>>> import chainer
>>> chainer.__version__
'1.0.0'

PEP 0396 -- Module Version Numbers
https://www.python.org/dev/peps/pep-0396/

pip install chainer-cuda-deps is recommended, and chainer-cuda-requirements is deprecated now. It will be removed in the future minor release.

Following result is at least not intuitive.

>>> a = Variable([1,2,3,4])
>>> b = np.array([4,3,2,1])
>>> a > b
array([ True,  True,  True,  True], dtype=bool)

>>> a = Variable([1,2,3,4])
>>> b = Variable([4,3,2,1])
>>> a > b
False

We should determine operations to be defined and what are desired result of them

When I generate document on Read The Docs, build task for PDF-formatted document returns 1 and fails to generate the document.

When CUDA is not enabled, it fails.
related to #7

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for Function, especially,

• Direct usage of Function.forward and Function.backward
• Function.unchain
• Function.to_gpu
• Function.to_cpu
• Function.parameters
• Function.gradients
• Split.add_branch
• Split.backward

Function.to_gpu and FunctionSet.to_gpu involves array copying, while an Optimizer object does not know this event. Consequently, user must migrate parameters onto devices BEFORE passing them to the optimizer. Violating this rule does not emit any messages currently, which ends up to making the training code meaningless.

So how about show a warning message if FunctionSet.collect_parameters is called after FunctionSet.to_gpu?

Current implementation of Linear and Convolution2D use their own parameter arrays. It lacks flexibility of the architecture design. Non-parameterized Linear is a fully-connected layer that uses some of input variables as its parameter. It should look like:

x, W, b = Variable(...), Variable(...), Variable(...)
y = F.linear(x, W=W, b=b)

Non-parameterized Convolution2D should have similar interface.

Related to #5

As discussed in #63, we open new issues about implementation of unit tests separately for each remaining core modules.

This issue is to add unittest for cuda package.

Use Python as ptb example.

Candidates I suppose are Caffe, Torch7, and Theano-based frameworks (like Pylean2, keras etc.).