mesnilgr / is13 Goto Github PK

Hello,

it would be really useful to know what are the elements on the .pkl.

There are 3 "test_set", 3 "train_set" and 3 dictionaries inside "dicts":

• train/test_set[1] is full of 0's and I dont have a clue about what they mean, would someone explain me what those are? (Are those the weight
  weight for the "meanings" of the words or something like that???)
• train/test_set[2] is probably the array of IOB-labels, since it has plenty of '126', and train/test_set[0] seems to be the word indexes arrays, because of the variety of numbers, right?
• dicts[0] is labels2idx, dict[2] is words2idx, but what is dicts[1], seems to??
• can I "invert" the dictionaries? Example: using labels2idx tuples "k : v" to build idx2labels as "v : k"?

I know they are simple questions, and I'm sorry if I'm being inconvenient with all those questions, but please be patient with me, I'm new on deeplearning. Thank you.

Ok, changed the download URL, and deleted the '.gz' from the atis.pkl files, but I'm still getting

File "/Users/lucaslourenco/Developer/is13-master/data/load.py", line 33, in atisfold
train_set, valid_set, test_set, dicts = cPickle.load(f)
IOError: Not a gzipped file

what method should I use instead of cPickle.load(f)?

def load(filename):
if not isfile(filename):
download('https://www.dropbox.com/s/3lxl9jsbw0j7h8a/'+filename)
f = gzip.open(filename,'rb')
return f

def atisfull():
f = load(PREFIX + 'atis.pkl')
train_set, test_set, dicts = cPickle.load(f)
return train_set, test_set, dicts

def atisfold(fold):
assert fold in range(5)
f = load(PREFIX + 'atis.fold'+str(fold)+'.pkl')
train_set, valid_set, test_set, dicts = cPickle.load(f)
return train_set, valid_set, test_set, dicts

thats how load.py is right now, there's something else I should change? Thank you for the alternative and the support @hadyelsahar and @mesnilgr!

as mentioned (on http://deeplearning.net/tutorial/rnnslu.html):
Running experiments on ATIS using this repository will run one epoch in less than 40 seconds

this must be for nh=100, right?
for nh=200 (at least for my pc) it takes about 3 times longer than for nh=100

--is that what it is supposed to be?
--how can we speed it up?

thanks in advance

Unable to run conlleval.pl even after downloading to code directory.
Errors :
File "C:\AppData\Local\Continuum\anaconda3\lib\subprocess.py", line 1178, in _execute_child
startupinfo)

OSError: [WinError 193] %1 is not a valid Win32 application

File "C:\AppData\Local\Continuum\anaconda3\lib\subprocess.py", line 1178, in _execute_child
startupinfo)

FileNotFoundError: [WinError 2] The system cannot find the file specified

Hi,
I'm running your code rnnslu.py for ATIS (the the 5 folds of data you provide us with)
(DeepLearningTutorials/code/rnnslu.py)

I expected to get F1=94.98% as mentioned in the paper:
'Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding'
(table 3: Elman/Single,context(9))

Instead, the best performance with this code is F1=94.058% for cont_win=9 and emb_dim=100.
Is that what I'm suppose to get?

Thank you in advance!

I've seen different issue reports from other people but they couldn't help me. I'm using windows now and no matter how I try, it seems I can't fix this, even with the path added to my environment variables. I've tried adding the is13 path in most different ways without success. I always end up getting "no module named is13.data" when try to run any of the two examples. Can someone help me? Thanks.

README.md - bug

git clone [email protected]:mesnilgr/is13.git
cd is13
python examples/elman-forward.py

error
Traceback (most recent call last):
File "examples/elman-forward.py", line 9, in
from is13.rnn.elman import model
ImportError: No module named is13.rnn.elman

git clone [email protected]:mesnilgr/is13.git
python is13/examples/elman-forward.py

is no error

In W7 It starts to learn on epoch 0, but then:

chmod('conlleval.pl', stat.S_IRWXU) # give the execute permissions

any hints? Thanks!

To examine the running of the code step-by-step, I copy-pasted all the elman-keras code in an iPython Notebook (with all the dependencies).
The weights were initialized normally after model.compile(..), but after running the training section, all the weights turned into 'nan'.
Need some help to understand the anomaly.
Thanks!

is13_elman.txt

Apologies for what is more a theoretical question than a code issue, but I am new to RNNs and am using this code to learn.

Why is the cost function based on the negative log likelihood of the last word in the sentence, rather than the mean of the NLLs of the entire sentence? I see this in line 53 of rnn/elman.py for examples.

Thanks in advance for the help as I learn.

hi,@mesnilgr. i want to run the code:

import theano, numpy
from theano import tensor as T

# nv :: size of our vocabulary
# de :: dimension of the embedding space
# cs :: context window size

nv, de, cs = 1000, 50, 5
emb = theano.shared(0.2 * numpy.random.uniform(-1.0, 1.0, \
(nv+1, de)).astype(theano.config.floatX)) # add one for PADDING at the end
idxs = T.imatrix() # as many columns as words in the context window and as many lines as words in the sentence
x = self.emb[idxs].reshape((idxs.shape[0], de*cs))

but errors happened in emb[idxs].i do not understand why emb[idxs] is correct?i think emb is a matrix,idxs is also a matrix.So why?

I would appreciate it if you could give me some tips.
thanks.

README.md -
The two lines of code to run the examples do not work if you do not add your actual path into the PYTHONPATH variable.

git clone [email protected]:mesnilgr/is13.git
python is13/examples/elman-forward.py

Will throw the error:

Traceback (most recent call last):
  File "./is13/examples/elman-forward.py", line 8, in <module>
    from is13.data import load
ImportError: No module named is13.data

By default Python do not add your actual path in sys.path and you need to add it explicitly.
In case of Bash I solved the problem by running:

git clone [email protected]:mesnilgr/is13.git
export PYTHONPATH=${PYTHONPATH}:`pwd -P`
python is13/examples/elman-forward.py

This line adds your current path without following symbolic links.

I know that in every Shell and Operating System the solution can be a different procedure, for that reason, I think you could just indicate that it is necessary to add your actual path in the variable PYTHONPATH.

Hello,

I'm trying to use your pre-processing methods to feed my LSTM RNN, and I cannot find a way to use the method for creating the Word Embeddings matrixes:

import theano, numpy
from theano import tensor as T

# nv :: size of our vocabulary
# de :: dimension of the embedding space
# cs :: context window size

nv, de, cs = 1000, 50, 5
embeddings = theano.shared(0.2 * numpy.random.uniform(-1.0, 1.0, \
(nv+1, de)).astype(theano.config.floatX)) # add one for PADDING at the end
idxs = T.imatrix() # as many columns as words in the context window and as many lines as words in the sentence
x = self.emb[idxs].reshape((idxs.shape[0], de*cs))

I can do the context window, but the code provided for the Word Embedding generation gives me this error:

NameError: name 'self' is not defined

I've tried to put the code in another .py file but is of no use, I'm sure it has a simple solution but I can't figure it out. Would you please give me a hand in this?

Hello！
Very nice work.
Can u give a valid link for this dataset? Thank u very much!

Can anyone provide a link for dataset download. The one given does not contain the file.

Thank you really much for offering your code, i appreciate it
(it's really helpful in order to understand rnns in slu - ive just started my master about them)

there are a couple of issues i've already dealt with in order to run rnnslu.py

I downloaded DeepLearningTutorials-master
and in code/rnnslu.py i replaced atisfold() and load() and added download_dropbox()

and finally i got the right data but now the error is
<<IndexError: index 557 is out of bounds for size 534>>
it is from function train() when self.sentence_train(words, labels, learning_rate)

as i see voc_size= 533 but here its trying to find the word id 557

SO: is it possible to upload sth that works without any errors?

(i tried to fix it myself, but then sth else came up and i thought maybe you could help)

thx in advance

Dir Sir or Madam,

The following errors occurred while running your code "is13/examples/elman-keras.py". Could you help me with that?

Thank you.

Traceback (most recent call last):
File "is13/examples/elman-keras.py", line 77, in
model.train_on_batch(X, Y)
File "/home/kevin/.local/lib/python2.7/site-packages/keras/models.py", line 1048, in train_on_batch
class_weight=class_weight)
File "/home/kevin/.local/lib/python2.7/site-packages/keras/engine/training.py", line 1838, in train_on_batch
self._make_train_function()
File "/home/kevin/.local/lib/python2.7/site-packages/keras/engine/training.py", line 997, in _make_train_function
**self._function_kwargs)
File "/home/kevin/.local/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1233, in function
return Function(inputs, outputs, updates=updates, **kwargs)
File "/home/kevin/.local/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1219, in init
**kwargs)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1841, in orig_function
fn = m.create(defaults)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1715, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/vm.py", line 1084, in make_all
impl=impl))
File "/home/kevin/.local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 866, in make_thunk
on_unused_input='ignore')
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function
output_keys=output_keys)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1841, in orig_function
fn = m.create(defaults)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1715, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/vm.py", line 1084, in make_all
impl=impl))
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/op.py", line 955, in make_thunk
no_recycling)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/op.py", line 858, in make_c_thunk
output_storage=node_output_storage)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1217, in make_thunk
keep_lock=keep_lock)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1157, in compile
keep_lock=keep_lock)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1620, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cmodule.py", line 1174, in module_from_key
module = lnk.compile_cmodule(location)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1523, in compile_cmodule
preargs=preargs)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cmodule.py", line 2368, in compile_str
return dlimport(lib_filename)
File "/home/kevin/.local/lib/python2.7/site-packages/theano/gof/cmodule.py", line 302, in dlimport
rval = import(module_name, {}, {}, [module_name])
ImportError: ('The following error happened while compiling the node', forall_inplace,cpu,scan_fn}(Shape_i{1}.0, Subtensor{int64:int64:int8}.0, IncSubtensor{InplaceSet;:int64:}.0, Shape_i{1}.0, simple_rnn_1/kernel, simple_rnn_1/recurrent_kernel, InplaceDimShuffle{x,0}.0), '\n', 'The following error happened while compiling the node', Dot22(<TensorType(float32, matrix)>, simple_rnn_1/kernel_copy), '\n', '/home/kevin/.theano/compiledir_Linux-3.8--generic-x86_64-with-Ubuntu-12.04-precise-x86_64-2.7.3-64/tmpRc1mHB/b8e34ea4363b3812bd111a731cb835f81ccc84235ec10367467e6198f7abc278.so: undefined symbol: _gfortran_st_write_done', '[Dot22(<TensorType(float32, matrix)>, simple_rnn_1/kernel_copy)]')

I wondering where to download an ATIS dataset including the intent labels?
The 5 folds available at

http://lisaweb.iro.umontreal.ca/transfert/lisa/users/mesnilgr/atis/

does not include the intent labels, and the dropbox link seems to be no longer active.

I would be very grateful for any hint of how to obtain the ATIS dataset.

/home/ubgpu/github/is13:/usr/local/lib/python2.7/dist-packages
ubgpu@ubgpu:~/github/is13$

ubgpu@ubgpu:/github/is13$ python examples/elman-forward.py
Traceback (most recent call last):
File "examples/elman-forward.py", line 8, in
from is13.data import load
ImportError: No module named is13.data
ubgpu@ubgpu:/github/is13$

In OSX, after finishing training epoch 0 I get this error:

OSError: [Errno 2] No such file or directory: 'conlleval.pl'

no acces to data on http://www-etud.iro.umontreal.ca/~mesnilgr/ anymore

no possible download of http://www-etud.iro.umontreal.ca/~mesnilgr/atis/atis.fold3.pkl.gz
and the perl evaluation script

cc @lucas0

petrux@PC:~$ git clone [email protected]:mesnilgr/is13.git
Cloning into 'is13'...
remote: Counting objects: 94, done.
remote: Total 94 (delta 0), reused 0 (delta 0), pack-reused 94
Receiving objects: 100% (94/94), 19.61 KiB | 0 bytes/s, done.
Resolving deltas: 100% (38/38), done.
Checking connectivity... done.

petrux@PC:~$ cd is13/
petrux@PC:~/is13$ ls
data  examples  __init__.py  metrics  README.md  rnn  utils

petrux@PC:~/is13$ python examples/jordan-forward.py 
Traceback (most recent call last):
  File "examples/jordan-forward.py", line 8, in <module>
    from is13.data import load
ImportError: No module named is13.data

Maybe there is something wrong with some __init__.py file? HTH.

EDIT: the problem was that the package was not in the PYTHONPATH. My bad. Sorry!

Hi

I am trying to use your code for event detection as a sequence labelling task. I looked at the atis dataset and tried to create one in same format but I get this error while train

ing. Could you please take a look at the pickle file attached. It just has one sentence in all training, test and validation sets for easier debugging.

I would really appreciate your help on this. I have been trying to get this done for quite sometime.

Traceback (most recent call last):
File "/home/ddua/workspace/RNN/examples/elman-forward.py", line 74, in
rnn.train(word_batch, label_last_word, s['clr'])
File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/function_module.py", line 615, in call

storage_map=self.fn.storage_map)

File "/usr/local/lib/python2.7/dist-packages/Theano-0.7.0-py2.7.egg/theano/compile/function_module.py", line 604, in call
outputs = self.fn()
IndexError: index out of bounds
Apply node that caused the error: Subtensor{int64, int64, int64}(forall_inplace,cpu,scan_fn}.1, Constant{-1}, Constant{0}, ScalarFromTensor.0)
Inputs types: [TensorType(float64, (False, True, False)), Scalar(int64), Scalar(int64), Scalar(int64)]
Inputs shapes: [(1, 1, 4), (), (), ()]
Inputs strides: [(32, 32, 8), (), (), ()]
Inputs values: [array([[[ 0.30284221, 0.21770984, 0.31160879, 0.16783916]]]), -1, 0, 5]

P.S. Please remove .png from the extension of pkl file. I couldn't attach a .pkl file in github.

Thanks a lot,
Dheeru Dua

Hi,

I would like to re-use the learned word embeddings (self.emb) in other experiments. It looks difficult to me to know what each word embedding (stored embeddings.npy) in the matrix corresponds to (words2idx dictionary). In addition there is an additional "word" which is used for padding.

My question is: how can I align the words with learned embeddings?

Thanks

i wanted to apply CRF over the RNN output of this repo following http://research.microsoft.com/apps/pubs/default.aspx?id=210167 . I need to replace the softmax function with the crf classsification .But finding difficult to implement in the recurrence function. Any help is appreciated.

Thanks

I'm not sure if data related issues are within scope, but the second example from the training dataset seems incorrect to me. The output looks like

                               WORD                               LABEL
                               what                                   O
                                 is                                   O
                                the                                   O
                            arrival                       B-flight_time
                               time                       I-flight_time
                                 in                                   O
                                san                 B-fromloc.city_name
                          francisco                 I-fromloc.city_name
                                for                                   O
                                the                                   O
                    DIGITDIGITDIGIT                  B-depart_time.time
                                 am                  I-depart_time.time
                             flight                                   O
                            leaving                                   O
                         washington                 B-fromloc.city_name

Shouldn't san francisco be toloc.city_name?

@mesnilgr @petrux @vierja @stray-leone
I have been reading the rnn code and I got an question in rnn/elman.py:
38--> def recurrence(x_t, h_tm1):
h_t = T.nnet.sigmoid(T.dot(x_t, self.Wx) + T.dot(h_tm1, self.Wh) + self.bh)
s_t = T.nnet.softmax(T.dot(h_t, self.W) + self.b)
return [h_t, s_t]

    [h, s], _ = theano.scan(fn=recurrence, \
        sequences=x, outputs_info=[self.h0, None], \
        n_steps=x.shape[0])

    p_y_given_x_lastword = s[-1,0,:]

It defines a simple recurrent network and my question is what's the component of s_t?
When we want to get p_y_given_x_lastword we use s[-1,0,:], "-1" means the lase element and ":" means
all, so why we use "0"? Because I think x_t and s_t is 1-d vector so why use index 0?
Could you solve this problem for me? Thank you very much.