monikkinom / ner-lstm Goto Github PK

Hi, thanks for your great work! I have a question about the cost function: Why do you define a cost function instead of the "categorical_crossentropy"? and, what is the difference between both of them?

Problem with running Hindi portion of code, ICON 2013 not CoNLL, Please give suggestions on how to proceed

First of all thanks so much for publishing this model, it is tremendously helpful.

One small suggestion: it takes a while to understand the purpose of the variable used (line 18). A more descriptive name for future readers may be something like words_used_in_sentence.

In model.py, I don't figure out the function of words_used_in_sent? Would you explain to me please? Thank U very much!

Error while running glove_model.py file

I am supplying a corpus to train, I have Copied corpus in Glove-1.2 directory and ran the code from embeddings folder and gave the name of corpus as param But I get the following error,

Stacktrace :
sudo python glove_model.py --corpus /home/swathimithran/workplace/NER-ISTM/ner-lstm-master/thirdparty/GloVe-1.2/english.txt --dimension 111 --glove_path /home/swathimithran/workplace/NER-ISTM/ner-lstm-master/thirdparty/GloVe-1.2/

mkdir -p build
gcc src/glove.c -o build/glove -lm -pthread -Ofast -march=native -funroll-loops -Wno-unused-result
gcc src/shuffle.c -o build/shuffle -lm -pthread -Ofast -march=native -funroll-loops -Wno-unused-result
gcc src/cooccur.c -o build/cooccur -lm -pthread -Ofast -march=native -funroll-loops -Wno-unused-result
gcc src/vocab_count.c -o build/vocab_count -lm -pthread -Ofast -march=native -funroll-loops -Wno-unused-result
BUILDING VOCABULARY
Processed 141 tokens.
Counted 103 unique words.
Truncating vocabulary at min count 5.
Using vocabulary of size 4.

COUNTING COOCCURRENCES
window size: 15
context: symmetric
max product: 13752509
overflow length: 38028356
Reading vocab from file "vocab.txt"...loaded 4 words.
Building lookup table...table contains 17 elements.
Processed 141 tokens.
Writing cooccurrences to disk..2 files in total.
Merging cooccurrence files: processed 17 lines.

SHUFFLING COOCCURRENCES
array size: 255013683
Shuffling by chunks: processed 17 lines.
Wrote 1 temporary file(s).
Merging temp files: processed 17 lines.

TRAINING MODEL
Read 17 lines.
Initializing parameters...done.
vector size: 111
vocab size: 4
x_max: 10.000000
alpha: 0.750000
demo.sh: line 35: 11029 Segmentation fault (core dumped) $BUILDDIR/glove -save-file $SAVE_FILE -threads $NUM_THREADS -input-file $COOCCURRENCE_SHUF_FILE -x-max $X_MAX -iter $MAX_ITER -vector-size $VECTOR_SIZE -binary $BINARY -vocab-file $VOCAB_FILE -verbose $VERBOSE
Traceback (most recent call last):
File "glove_model.py", line 48, in
model = GloveVec(args)
File "glove_model.py", line 21, in init
for line in open(path):
IOError: [Errno 2] No such file or directory: 'vectors.txt'

Great work. Can you share and desbribe your work @monikkinom.
Many Thank.

How can we use it in production. Lets say I made the model here and stored its object. Now in production when i get user's query, do we have to run tf.Session() 's run every time?

Hello guys,

Thank you for the great work

I have been trying to adopt your code to my work but get this error

ValueError: Cannot feed value of shape (128, 40, 207) for Tensor 'Placeholder:0', which has shape '(?, 40, 200)'

I am not an ML guru but have tried to follow the instructions as best I can. What could be the issue?
My embedding size is 200, sentence length is 40 and batch size is 128.

Hey @monikkinom!

Cool work. Was this implementation the result of some paper? If so, what was it?

I wanted to print the prediction of individual word of an unclassified text how should I do it..?

WARNING:tensorflow:From model.py:99 in train.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.
Traceback (most recent call last):
File "model.py", line 146, in
train(parser.parse_args())
File "model.py", line 99, in train
model = Model(args)
File "/Users/kumarnitin/Code/chanakya/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/Users/kumarnitin/Code/chanakya/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 943, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (4, 30, 111) for Tensor u'Placeholder:0', which has shape '(?, 1000, 100)'

Traceback (most recent call last):
File "get_conll_embeddings.py", line 130, in
sentence_length=args.sentence_length)
File "get_conll_embeddings.py", line 92, in get_input
temp = model[line.split()[0]]
KeyError: 'EU'

i am getting this error ,while trying to generate embeddings.please guide me!

First of all work is great and inspirational with very good results for a Bidirectional LSTM. Thanks for making this available! I know you have written very nice readme file but still, I am finding it little difficult to run it on my local system, specifically generating the train_embed.pkl, train_tag.pkl , test_a_embed.pkl and test_a_tag.pkl file. Your help will be much appreciated if you can help me in generating this embeddings files with sample input text and default parameters.

Hi! I would like to play with this deep learning model for NER with some data. Is there anywhere I could know the version that this model is using for python, tensorflow or any libraries? I try to search for configurations for these in README but did not find any.

It will be really helpful, If you can provide a link to download hindi training data.

wwhen i run rsizeinput file oupput file is empty . why it is

I'm using Tensorflow 1.0 and I got an error message as below while running model.py.

AttributeError: module 'tensorflow.python.ops.nn' has no attribute 'bidirectional_rnn'

I looked up this problem in StackOverflow, and users told me that:

In TensorFlow 1.0, you have the choice of two bidirectional RNN functions:
tf.nn.bidirectional_dynamic_rnn()
tf.contrib.rnn.static_bidirectional_rnn()

Any idea which bidirectional RNN should I change to?

Hi,author
i am studying this code, but i try to run this project with some times ,always faild , can you write how to run this project in README.md ,thx!
best wishes!

@monikkinom
I get a error when run main.py
ImportError: This module is deprecated. Use tf.nn.rnn_cell instead.
My version Tensor Flow is 0.9; Do you have any idea to solve that?
Many thanks

Is there any how to create a file that will have the list trained NERs from the corpus that we ingested?

hello sir.
you have did great work to post this code.
but i am getting this error.
please help me!
File "model.py", line 123, in
train(parser.parse_args())
File "model.py", line 81, in train
train_inp, train_out = get_train_data()
File "/home/narvin123/ner-lstm-master/input.py", line 6, in get_train_data
emb = pickle.load(open('embeddings/train_embed.pkl', 'rb'))
IOError: [Errno 2] No such file or directory: 'embeddings/train_embed.pkl'

why should test data in .iob format? it should be sentences or tokens instead of tagged data.

Hi, I have few doubts in the code, perhaps some of these would be really silly. But I'll really appreciate if someone can explain them.

1. In glove_model.py, what is vectors.txt?

parser.add_argument('--restore', type=str, default=None, help='pre-trained glove vectors.txt')

2. In model.py, what does class_size refers to and its implication?

parser.add_argument('--class_size', type=int, help='number of classes', required=True)

3. What does the following output of model.py imply?

[nan, 1.0, nan, nan, 0.90909090909090906, 0.5]
max model saved in file: model_max.ckpt
test_b score:
[nan, 0.40000000000000002, nan, nan, 0.76190476190476197, 0.14814814814814814]

4. In sample.py, what is input_embed?

parser.add_argument('--input_embed', type=str, help='location of input pickle embedding', required=True)

I'm a newbie in this particular field. Any help would be much appreciated.
I'm getting the following error.

Traceback (most recent call last):
  File "model.py", line 124, in <module>
    train(parser.parse_args())
  File "model.py", line 96, in train
    model.output_data: train_out[ptr:ptr + args.batch_size]})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 944, in _run
    % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (4, 1000, 122) for Tensor u'Placeholder:0', which has shape '(?, 1000, 111)'

The args I've set in are as follows:

python wordvec_model.py --corpus 'all_data.txt' --dimension 111 --vocab_size 5000000 --sentence_length 1000 --workers 4

python get_conll_embeddings.py --train 'eng.train_50' --test_a 'eng.test_a_50' --test_b 'eng.test_b_50' --sentence_length 1000 --use_model 'wordvec_model_111.pkl' --model_dim 111

python model.py --word_dim 111 --sentence_length 1000 --class_size 20 --batch_size 256

I'm getting the error in this part.
Before execution our objs look like this:

model.input_data: <tf.Tensor 'Placeholder:0' shape=(?, 1000, 111) dtype=float32>
model.output_data: <tf.Tensor 'Placeholder_1:0' shape=(?, 1000, 20) dtype=float32>

sess.run(model.train_op, {model.input_data: train_inp[ptr:ptr + args.batch_size],
                                          model.output_data: train_out[ptr:ptr + args.batch_size]})

Also, I ported to tensorflow 1.0, but that shouldn't be a problem here I guess.

Hi, I was reported the following error, Anyone with me? And how to resolve it ?
Thank you so much.

Traceback (most recent call last):
File "model.py", line 122, in
train(parser.parse_args())
File "model.py", line 94, in train
model.output_data: train_out[ptr:ptr + args.batch_size]})
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 938, in _run
np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
File "/usr/lib64/python2.7/site-packages/numpy/core/numeric.py", line 532, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.

how to generate training data to train our custom ner ? Can any one help ?

While running wordvec_model.py file,
Which file should I pass as corpus file :- parser.add_argument('--corpus', type=str, help='corpus location', required=True) ?

when i run this command
python resize_input.py --input JAMSHEDPUR00.text --output output1 --trim 50

i got this output it displays as:

0 sentences trimmed out of 0 total sentences

This is some interesting work, with very good results for a Bidirectional LSTM. Thanks for making this available! Can you describe your recommend process of setting up your experiment?

when i run this ner on tensorflow1.3，i had got many errors。can you help?

In ner-lstm/models.py, lines 13-14, shouldn't there be a placeholder instead of the numerical value
because one would like to set the output_keep_prob to 1.0 at the inference time?

for 'get_conll_embeddings.py', it is saying that I have to input locations of the train, test_a, test_b, and the use_model locations.
Train, Test_a and Test_b are the txt files that I can copy from the data folder, correct?
and use_model is the pkl file that I created using the wordvec or other embedding model, correct?

If this is the case, how am I truly training the model with the corpus that I ingested from the start?
Or does that means I have to make my own train, test_a and test-b dataset?

We execute the code on CoNLL 2003 English dataset using Word2Vec 311 dimensions, but we got the results of 83.68%(Test_a) and 79.68%(Test_b), about 10% lower than the report.