zhangxiangxiao / crepe Goto Github PK

Character-level Convolutional Networks for Text Classification

License: BSD 3-Clause "New" or "Revised" License

Lua 100.00%

crepe's Introduction

Crepe

This repository contains code in Torch 7 for text classification from character-level using convolutional networks. It can be used to reproduce the results in the following article:

Xiang Zhang, Junbo Zhao, Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015)

Note: An early version of this work entitled “Text Understanding from Scratch” was posted in Feb 2015 as arXiv:1502.01710. The present paper above has considerably more experimental results and a rewritten introduction.

Note: See also the example implementation in NVIDIA DIGITS. Using CuDNN is 17 times faster than the code here.

Components

This repository contains the following components:

data: data preprocessing scripts. It can be used to convert csv format to a Torch 7 binary format that can be used by the training program directly. We used csv format to distribute the datasets in our article. The datasets are available at http://goo.gl/JyCnZq.
train: training program.

For more information, please refer to the readme files in each component directory.

Example Usage

Here is an example of using our data tools and training programs pipeline to replicate the small convolutional network for DBPedia ontology classification in the article. First, clone the project and download the file dbpedia_csv.tar.gz from our storage in Google Drive to the data directory. Then, uncompress the files and build t7b files using our dataset tools.

$ cd data
$ tar -xvf dbpedia_csv.tar.gz
$ qlua csv2t7b.lua -input dbpedia_csv/train.csv -output train.t7b
$ qlua csv2t7b.lua -input dbpedia_csv/test.csv -output test.t7b
$ cd ..

In the commands above, you can replace qlua by luajit as long as it has an associated torch 7 distribution installed. Now there will be 2 files train.t7b and test.t7b in the data directory. Normally, the second step is to go to the train directory and change the configurations listed in config.lua, especially for data file location and number of output units in the last linear layer. This last linear layer is important because its number of output units should correspond to the number of classes in your dataset. Luckily for this example on DBPedia ontology dataset the configurations are all set. One just needs to go into the train directory and start the training process

$ cd train
$ qlua main.lua

This time we have to use qlua, because there is a nice visualization using Qt that is updated for every era. Please make sure packages qtlua and qttorch are installed in your system and there is a corresponding X to your terminal. To run this example successfully you will also need a NVidia GPU with at least 3GB of memory. Otherwise, you can configure the model in train/config.lua for less parameters.

Okay! If you start to find out checkpointing files like main_EPOCHES_TIME.t7b and sequential_EPOCHES_TIME.[t7b|png] appearing under the train directory in several hours or so, it means the program is running without problems. You should probably find some other entertainment for the day. :P

Issues

It is discovered that the alphabet actually has two hyphen/minus characters ('-'). This issue was present for the results in the papers as well. Since this is probably a minor issue, we will keep the alphabet configurations as is to ensure reproduceability. That said, you are welcome to change the alphabet variable in train/config.lua to remove it. See issue #4 for more details.

Why Call It "Crepe"?

It is just a word popping up to my mind pondering for a repository name in Github. It has nothing to do with French cuisine, text processing or convolutional networks. If a connection is really really needed, how about "Convolutional REPresentation of Expressions"?

crepe's People

Contributors

Stargazers

Watchers

Forkers

silky beronx86 sxhfut nickyuan weixiaohua amos-zq superneo provemyself saikrishnar lizhangzhan binqilin cfandy hengqujushi yliuhb craftsliu chagge zhoujialinmumu detrident narayana1208 thanujadax solualexis memkite daishichao renyuan1010 chiahungtai snazz2001 wgmueller1 kuyezhiying chenglongchen edran jsalim jakubzitny jethrotan vamsijkrishna yanshanjing mokacao cojito javaosos wwhu yanqingmen samuel2015 devsinghsachan yanweifu he0x shyamalschandra arpitjain2811 vikkamath mlstudy jjwangnlp vseledkin jakezhaojb albertofernandezvillan peterjliu shuggiefisher alex2awesome vijayendra-g easycui eunice-33 lu839684437 viirya npow stefanziv salemameen cheng6076 enjoyhacking liumangtu coreyt aganeshlab41 le02146 52nlp wmk26 khellan superchuck atrus7 oztc mquad ironj eden57 zhj12388 zbxzc35 bogger christiansch liulinhere milestonesvn dzenteno2 marathonfail hitluobin liangkai sherkwast caomw iij0 dxl0632 shtprogrammer kuibu hughperkins pengsun wandabwa2004 zayedmohamed schrodinger1926 kalyanp

crepe's Issues

qlua: ./model.lua:237: inconsistent tensor size at /home/ubuntu/torch/pkg/torch/lib/TH/generic/THTensorCopy.c:7

I am getting this error while running with cudnn on dbpedia dataset. Any help?

error running example code

When I run $ qlua main.lua, I get the following error message:

dyld: Library not loaded: @rpath/./libQtGui.4.dylib
Referenced from: /usr/local/bin/qlua
Reason: image not found
Trace/BPT trap: 5

My environment: os x 10.10.5, torch7, lua 5.1.4, xcode 6.2

Testing on training data for era 1 never stops

It seems that the part "Testing on training data for era 1 never stops" doesn't stop on the example data.
The function Test:run(logfunc) starts a loop. I have no idea how to control this loop.
What parameters should change in config.lua?

Thanks in advance!

Question on batch normalization.

Why did you not use batch normalization on the Crepe network described in Text Understanding from Scratch? I looked at the paper but couldn't find anything that mentions it...
Does it have to do with the sparsity of the input data?

Thanks a lot for your work!

Issue in saving data

The networks executes well but after completing all the era it get's stuck and just displays 'Saving data' and doesn't complete the execution.

Replicating on mxnet - too much memory for GPU

I'm having a bit of difficulty replicating this cool model using the mxnet package (instead of torch). I'm not sure if it's something to do with my implementation of it, or the mxnet package - however it takes up way more than 3GB of memory.

My version is here and below is a direct paste of the model:

Edit: fixed with updated params

    input_x = mx.sym.Variable('data')  # placeholder for input
    input_y = mx.sym.Variable('softmax_label')  # placeholder for output
    #6 Convolutional layers
    #1. alphabet x 1014
    conv1 = mx.symbol.Convolution(
        data=input_x, kernel=(7, 69), num_filter=NUM_FILTERS)
    relu1 = mx.symbol.Activation(
        data=conv1, act_type="relu")
    pool1 = mx.symbol.Pooling(
        data=relu1, pool_type="max", kernel=(3, 1), stride=(1, 1))
    #2. 336 x 256
    conv2 = mx.symbol.Convolution(
        data=pool1, kernel=(7, 1), num_filter=NUM_FILTERS)
    relu2 = mx.symbol.Activation(
        data=conv2, act_type="relu")
    pool2 = mx.symbol.Pooling(
        data=relu2, pool_type="max", kernel=(3, 1), stride=(1, 1))
    #3. 110 x 256
    conv3 = mx.symbol.Convolution(
        data=pool2, kernel=(3, 1), num_filter=NUM_FILTERS)
    relu3 = mx.symbol.Activation(
        data=conv3, act_type="relu")
    #4. 108 x 256
    conv4 = mx.symbol.Convolution(
        data=relu3, kernel=(3, 1), num_filter=NUM_FILTERS)
    relu4 = mx.symbol.Activation(
        data=conv4, act_type="relu")
    #5. 106 x 256
    conv5 = mx.symbol.Convolution(
        data=relu4, kernel=(3, 1), num_filter=NUM_FILTERS)
    relu5 = mx.symbol.Activation(
        data=conv5, act_type="relu")
    #6. 104 x 256
    conv6 = mx.symbol.Convolution(
        data=relu5, kernel=(3, 1), num_filter=NUM_FILTERS)
    relu6 = mx.symbol.Activation(
        data=conv6, act_type="relu")
    pool6 = mx.symbol.Pooling(
        data=relu6, pool_type="max", kernel=(3, 1), stride=(1, 1))
    #34 x 256
    flatten = mx.symbol.Flatten(data=pool6)
    #3 Fully-connected layers
    #7.  8704
    fc1 = mx.symbol.FullyConnected(
        data=flatten, num_hidden=1024)
    act_fc1 = mx.symbol.Activation(
        data=fc1, act_type="relu")
    drop1 = mx.sym.Dropout(act_fc1, p=0.5)
    #8. 1024
    fc2 = mx.symbol.FullyConnected(
        data=drop1, num_hidden=1024)
    act_fc2 = mx.symbol.Activation(
        data=fc2, act_type="relu")
    drop2 = mx.sym.Dropout(act_fc2, p=0.5)
    #9. 1024
    fc3 = mx.symbol.FullyConnected(
        data=drop2, num_hidden=NOUTPUT)
    crepe = mx.symbol.SoftmaxOutput(
        data=fc3, label=input_y, name="softmax")

Attempt to call field 'TemporalConvolution_updateOutput' (a nil value)

I am using Crepe for my sequential data. On executing main.lua the following error appears:
qlua: ...b/torch/install/share/lua/5.1/nn/TemporalConvolution.lua:41: attempt to call field 'TemporalConvolution_updateOutput' (a nil value)
stack traceback:
[C]: in function 'TemporalConvolution_updateOutput'
...b/torch/install/share/lua/5.1/nn/TemporalConvolution.lua:41: in function 'updateOutput'
...e/user/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
./model.lua:30: in function 'forward'
./train.lua:89: in function 'batchStep'
./train.lua:68: in function 'run'
main.lua:140: in function 'run'
main.lua:42: in function 'main'
main.lua:316: in main chunk

Is it some kind of a bug or any specific change is to be done with the configuration?

Thanks,
Asif

Exception while testing on custom dataset

Hi,
I've created my own training and testing datasets with just 2 classes. I changed the number of output classes in config.lua and set it to 2 as per instructions in the related article. Now, the training goes on well, but testing crashes with this exception:

Testing on training data for era 1
qlua: /home/murad/torch/install/share/lua/5.1/nn/THNN.lua:110: Assertion `cur_target >= 0 && cur_target < n_classes' failed. at /home/murad/torch/extra/nn/lib/THNN/generic/ClassNLLCriterion.c:57
stack traceback:
[C]: at 0x7f0b7de7ef50
[C]: in function 'v'
/home/murad/torch/install/share/lua/5.1/nn/THNN.lua:110: in function 'ClassNLLCriterion_updateOutput'
...rad/torch/install/share/lua/5.1/nn/ClassNLLCriterion.lua:44: in function 'forward'
./test.lua:60: in function 'run'
main.lua:145: in function 'run'
main.lua:42: in function 'main'
main.lua:316: in main chunk

I googled it out and read that it means that the number of output classes in the last layer doesn't match the number of classes in dataset which obviously cannot be, I checked it out stupid amount of times!
I'm running the code without CUDA, in CPU only mode. Will appreciate any help. Thank you.

[Question] Would using ReLU instead of Threshold produce similar results?

Hi, I've noticed that in the model you are using Threshold instead of ReLU. What is the reason behind this choice?

If I'm reading the source correctly, the only difference between those is that Thresold has self.threshold initialized at 1e-6 instead of 0 for ReLU [1]. Is this affecting the success of the model in a significant way?

[1] https://github.com/torch/nn/blob/master/ReLU.lua

New dataset location?

Is there a new location for the dataset? it appears the link is broken

duplicate hyphen (minus?) in alphabeta?

In train/config.lua, there are two hyphens (minus) in alphabeta:
... 9-, ...
and
... +-= ...

Is it intentional? If not, does it affect the convnet size specs?

Unable to connect X11 server

When I run the main.lua by 'qlua main.lua', I got bugs like below:

Unable to connect X11 server (continuing with -nographics)
Device set to 1
Loading datasets...
Loading the model...
Model randomized.
Current model type: torch.CudaTensor
Loading the trainer...
Loading the tester...
qlua: not loading module qtuiloader (running with -nographics)
qlua: not loading module qtgui (running with -nographics)
qlua: qtwidget window functions will not be usable (running with -nographics)
qtwidget window functions will not be usable (running with -nographics)
qlua: ./scroll.lua:16: attempt to index global 'qtuiloader' (a nil value)
stack traceback:
    [C]: in function '__index'
    ./scroll.lua:16: in function '__init'
    ...ysfeng/lvchao/torch/install/share/lua/5.1/torch/init.lua:91: in function <...ysfeng/lvchao/torch/install/share/lua/5.1/torch/init.lua:87>
    [C]: in function 'Scroll'
    ./mui.lua:21: in function '__init'
    ...ysfeng/lvchao/torch/install/share/lua/5.1/torch/init.lua:91: in function <...ysfeng/lvchao/torch/install/share/lua/5.1/torch/init.lua:87>
    [C]: in function 'Mui'
    main.lua:123: in function 'new'
    main.lua:41: in function 'main'
    main.lua:316: in main chunk
'''
How can I remove the ui show?

yelp data set

one question regarding yelp,

from which year is this dataset? and where it from exactly

thanks!

Can I run this not on a GPU

Having a cpu machine, can I run Crepe or should I get a GPU AWS machine?

Thanks
-Yair

Not able to understant where you did preprocessing of text?

Hi, which module is responsible for preprocessing of input text?

config.val_data.batch_size is not defined

config.train_data.batch_size is defined twice at Line 18 and Line 25 in train/config.lua . Looks like the seconds one is supposed to be config.val_data.batch_size ?

How do I save the confusion matrix data in a vector?

I am looking to save the confusion matrix data as a vector (in test.lua) ...is there a way to save and load it in lua?
Also, the results reports in the paper...is 1.5% the error rate on the DBPedia dataset? I was unable to replicate the results so I was wondering...

How to run this in cpu only mode?

My gpu is not supported under cuda. Also is there an aws ami with this pre installed? And what would be the expected training time for dbpedia dataset on an aws g2xlarge?

Printing classes

I was told that I can used test.decision[i] to see the classes Crepe outputs.
Would I print out test.decision[i] in main.lua?
Also is there a way to associate the i-th input with test.decision[i]?
Can I do the same for training?

How to change Crepe configurations to TEST Dataset on Trained Model

I was wondering, how to use already trained model for validating my dataset; Can you please highlight all steps how to manage such change in crepe because -resume is not fulfilling my requirement;

Lastly, If I want to change error function to predict only one label, can you shed some light on that as well?

Regards,
Khan Awais

Training on a custom dataset

Hi Zhang,

Thanks for open sourcing this code. I am trying to run your model on my dataset, I have pre-processed the data in it's required format.

Now I am not familiar with lua and torch, so can you tell me is there anything I need to change in the config file to run it over a different dataset than dbpedia one? (I could only decipher the no of classes parameter from config file). Sorry if the question sounds elementary, I just want to make sure I am folllowing the right steps for my experiements.

Thanks in advance.

DIGITS Tutorial based on this project

Hi, just to let watchers know that I have added a Tutorial on DIGITS for text classification using the model from this project.

See the write-up

I am using CuDNN and an optimized data loader and training time is an order of magnitude faster than the reference implementation. On my system it takes ~15mn to train one epoch of 498400 training samples and 56000 validation samples, with 4 validation sweeps per epoch.

[Question] Where is batch Tensor initialized?

Sorry for the dumb question I must have missed something but I fail to see where the Tensor used to store batch data is initialized in this function? I can see how the one "bit" corresponding to each character is set in stringToTensor but don't you need to explicitly zero-initialize the tensor on that line?

Y-axis scale and units?

I just want to verify that my network is working correctly and that I'm interpreting the graphs correctly on my own data. I was tinkering around with the code before launching it, so not sure if I accidentally messed up somewhere. Several questions:

For loss, I got values over 1 for validation. How bad is that? Is a loss over 1 big or is it reasonable?
Is the error y-axis unit in percent? So here, for the error, would it range from 0% to 0.16% or is it 0% to 16%?
When I trained on the dbpedia_csv data, the loss and error graphs looked about the same shape. In general, should the loss and error graphs always have the same shape?
Are the error and loss on different scales?

Apologies again, Zhang, for all the questions.

Forward propagation time benchmark

Hi, may I know your forward propagation time in test.lua recorded here self.time.forward?

I would like to train the model on GPU machine (aws g2.8xlarge, CudaTensor), and test on large data set on cluster using CPU(DoubleTensor)only. In my case, using GPU the forward time is around 1.5s, and CPU is 3s.

I think it's very slow, may I know your time for comparison? so that I can check if there is some wrong with my setting.

Thanks a lot