roomylee / cnn-relation-extraction Goto Github PK

View Code? Open in Web Editor NEW

220.0 15.0 62.0 2.95 MB

Tensorflow Implementation of Convolutional Neural Network for Relation Extraction (COLING 2014, NAACL 2015)

Home Page: http://www.aclweb.org/anthology/W15-1506

License: MIT License

Perl 40.74% Python 59.26%

convolutional-neural-networks relation-extraction semeval-2010 tensorflow cnn-text-classification

cnn-relation-extraction's Introduction

Convolutional Neural Networks for Relation Extraction

Tensorflow Implementation of Deep Learning Approach for Relation Extraction Challenge(SemEval-2010 Task #8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals) via Convolutional Neural Networks.

Usage

Train

Train data is located in "SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT".
"GoogleNews-vectors-negative300" is used as pre-trained word2vec model.
Performance (accuracy and f1-socre) outputs during training are NOT OFFICIAL SCORE of SemEval 2010 Task 8. To compute the official performance, you should proceed the follow Evaluation step with checkpoints obtained by training.

Display help message:

$ python train.py --help

Train Example:

$ python train.py --embedding_path "GoogleNews-vectors-negative300.bin"

Evaluation

You can get an OFFICIAL SCORE of SemEval 2010 Task 8 for test data by following this step. README describes how to evaluate the official score.
Test data is located in "SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT".
MUST GIVE --checkpoint_dir ARGUMENT, checkpoint directory from training run, like below example.

Evaluation Example:

$ python eval.py --checkpoint_dir "runs/1523902663/checkpoints/"

Results

Officiail Performance

Learning Curve (Accuracy)

Learning Curve (Loss)

SemEval-2010 Task #8

Given: a pair of nominals
Goal: recognize the semantic relation between these nominals.
Example:
- "There were apples, pears and oranges in the bowl."
  → CONTENT-CONTAINER(pears, bowl)
- “The cup contained tea from dried ginseng.”
  → ENTITY-ORIGIN(tea, ginseng)

The Inventory of Semantic Relations

Cause-Effect(CE): An event or object leads to an effect(those cancers were caused by radiation exposures)
Instrument-Agency(IA): An agent uses an instrument(phone operator)
Product-Producer(PP): A producer causes a product to exist (a factory manufactures suits)
Content-Container(CC): An object is physically stored in a delineated area of space (a bottle full of honey was weighed) Hendrickx, Kim, Kozareva, Nakov, O S´ eaghdha, Pad ´ o,´ Pennacchiotti, Romano, Szpakowicz Task Overview Data Creation Competition Results and Discussion The Inventory of Semantic Relations (III)
Entity-Origin(EO): An entity is coming or is derived from an origin, e.g., position or material (letters from foreign countries)
Entity-Destination(ED): An entity is moving towards a destination (the boy went to bed)
Component-Whole(CW): An object is a component of a larger whole (my apartment has a large kitchen)
Member-Collection(MC): A member forms a nonfunctional part of a collection (there are many trees in the forest)
Message-Topic(CT): An act of communication, written or spoken, is about a topic (the lecture was about semantics)
OTHER: If none of the above nine relations appears to be suitable.

Distribution for Dataset

SemEval-2010 Task #8 Dataset [Download]

Relation	Train Data	Test Data	Total Data
Cause-Effect	1,003 (12.54%)	328 (12.07%)	1331 (12.42%)
Instrument-Agency	504 (6.30%)	156 (5.74%)	660 (6.16%)
Product-Producer	717 (8.96%)	231 (8.50%)	948 (8.85%)
Content-Container	540 (6.75%)	192 (7.07%)	732 (6.83%)
Entity-Origin	716 (8.95%)	258 (9.50%)	974 (9.09%)
Entity-Destination	845 (10.56%)	292 (10.75%)	1137 (10.61%)
Component-Whole	941 (11.76%)	312 (11.48%)	1253 (11.69%)
Member-Collection	690 (8.63%)	233 (8.58%)	923 (8.61%)
Message-Topic	634 (7.92%)	261 (9.61%)	895 (8.35%)
Other	1,410 (17.63%)	454 (16.71%)	1864 (17.39%)
Total	8,000 (100.00%)	2,717 (100.00%)	10,717 (100.00%)

Reference

Relation Classification via Convolutional Deep Neural Network (COLING 2014), D Zeng et al. [review] [paper]
Relation Extraction: Perspective from Convolutional Neural Networks (NAACL 2015), TH Nguyen et al. [review] [paper]
dennybritz's cnn-text-classification-tf repository [github]

cnn-relation-extraction's People

Contributors

Stargazers

Watchers

cnn-relation-extraction's Issues

A question about the testing result

Hi,
I tried to train the model by using SemEval2010_task8 data set, but the result is not as good as the paper 'Perspective from Convolutional Neural Networks', the F1-measure of my result is only 74%.
Can you tell me your result?
Thanks
Niko

deriving values for e1 e2

Hi , I was able to get the classification label , based on the approach you suggested , But i am not able to figure out how to get the relation label ?

i inverted the labelsmapping and was able to pass all_predictions in an iterator , to get the labels out .

for prediction in all_predictions: labelsMapping = { 0:'Other', 1 : 'Message-Topic(e1,e2)', 2 :'Message-Topic(e2,e1)', 3 : 'Product-Producer(e1,e2)', 4 :'Product-Producer(e2,e1)', 5 : 'Instrument-Agency(e1,e2)', 6 :'Instrument-Agency(e2,e1)', 7 : 'Entity-Destination(e1,e2)', 8 : 'Entity-Destination(e2,e1)', 9 : 'Cause-Effect(e1,e2)', 10 : 'Cause-Effect(e2,e1)', 11 : 'Component-Whole(e1,e2)', 12 :'Component-Whole(e2,e1)', 13 : 'Entity-Origin(e1,e2)', 14 :'Entity-Origin(e2,e1)', 15 : 'Member-Collection(e1,e2)', 16 : 'Member-Collection(e2,e1)', 17 : 'Content-Container(e1,e2)', 18 : 'Content-Container(e2,e1)'} print(labelsMapping[int(prediction)])

but i am not able to figure out how to get into the values for effect e1, e2 etc .. could you help me out with this ?

question in utils.py

initW = np.random.randn(len(vocab.vocabulary_), embedding_dim).astype(np.float32) / np.sqrt(len(vocab.vocabulary_))

why there exists the np.sqrt(len(vocab.vocabulary))?what is the aim?

eval accuracy

why the train model's accuracy is around 90% but the accuracy of eval just 60% ?

Chinese relation extra

Can this code extract Chinese relations?

problem in executing eval.py file

i am following instructions as per READ.me but following problem arises in running eval.py
Traceback (most recent call last):
File "/home/srinath/RE/cnn-relation-extraction-master/eval.py", line 122, in
tf.app.run()
File "/home/srinath/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/srinath/RE/cnn-relation-extraction-master/eval.py", line 118, in main
eval()
File "/home/srinath/RE/cnn-relation-extraction-master/eval.py", line 80, in eval
batches = data_helpers.batch_iter(list(x_eval), FLAGS.batch_size, 1, shuffle=False)
File "/home/srinath/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/flags.py", line 50, in getattr
raise AttributeError(name)
AttributeError: batch_size

Steps to run the model

May you send the steps to run the model?

problem in eval.py

MODEL OVERFITTING ! !

high training accuracy,low testing accuracy

which version of tensorflow should be?

ub16hp@UB16HP:~/ub16_prj/cnn-relation-extraction$ python3 train.py --embedding_path ../../data/GoogleNews-vectors-negative300.bin

DROPOUT_KEEP_PROB=0.5
BATCH_SIZE=20
NUM_FILTERS=128
TEXT_EMBEDDING_DIM=300
ALLOW_SOFT_PLACEMENT=True
MAX_SENTENCE_LENGTH=90
GPU_ALLOW_GROWTH=True
CHECKPOINT_DIR=
TRAIN_PATH=SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT
NUM_EPOCHS=100
POS_EMBEDDING_DIM=50
LEARNING_RATE=1.0
EMBEDDING_PATH=../../data/GoogleNews-vectors-negative300.bin
FILTER_SIZES=2,3,4,5
EVALUATE_EVERY=100
TEST_PATH=SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT
NUM_CHECKPOINTS=5
L2_REG_LAMBDA=1e-05
LOG_DEVICE_PLACEMENT=False
DESC=
DISPLAY_EVERY=10
DEV_SAMPLE_PERCENTAGE=0.1
DECAY_RATE=0.9

SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT
max sentence length = 89

Text Vocabulary Size: 19151
x = (8000, 90)
y = (8000, 19)

Position Vocabulary Size: 162
position_1 = (8000, 90)
position_2 = (8000, 90)

Train/Dev split: 7200/800

2018-10-31 10:52:56.457227: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-10-31 10:52:56.534546: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-31 10:52:56.534922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 950M major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 3.54GiB
2018-10-31 10:52:56.534949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0, compute capability: 5.0)
Traceback (most recent call last):
File "train.py", line 185, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 181, in main
train()
File "train.py", line 82, in train
l2_reg_lambda=FLAGS.l2_reg_lambda)
File "/home/ub16hp/ub16_prj/cnn-relation-extraction/text_cnn.py", line 64, in init
losses = tf.nn.softmax_cross_entropy_with_logits_v2(logits=self.logits, labels=self.input_y)
AttributeError: module 'tensorflow.python.ops.nn' has no attribute 'softmax_cross_entropy_with_logits_v2'
ub16hp@UB16HP:~/ub16_prj/cnn-relation-extraction$