slczgwh / redn Goto Github PK

I'm trying to use the finetuned model to predict relations in a new sentence, but I cannot find the instructions. Following the OpenNRE, I have tried to add the following code:

_framework.model.infer({"token": ["he", "has", "combined", "intelligent", "history", "with", "moving", "prose", ",", "creating", "an", "informative", ",", "inspiring", "narrative", "telling", "the", "history", "of", "a", "great", "people", "."], "h": {"pos": [14, 15]}, "t": {"pos": [17, 18]}})

But the program raised an error:

Traceback (most recent call last):
File "E:/Project/REDN-master/redn_trainer.py", line 195, in
train(**args_list[0])
File "E:/Project/REDN-master/redn_trainer.py", line 143, in train
_framework.model.infer({"token": ["he", "has", "combined", "intelligent", "history", "with", "moving", "prose", ",", "creating", "an", "informative", ",", "inspiring", "narrative", "telling", "the", "history", "of", "a", "great", "people", "."], "h": {"pos": [14, 15]}, "t": {"pos": [17, 18]}})
File "E:\Project\REDN-master\opennre\model\para.py", line 46, in infer
logits = self.forward(*item)
TypeError: forward() takes 3 positional arguments but 4 were given

Then I found the 45 line of "REDN-master\opennre\model\para.py" called the self.sentence_encoder.tokenize method and return indexed_tokens, att_mask, new_index, but the self.forward seems to need tokens, att_mask only, so I changed the line 46 to logits = self.forward(item[0], item[1]), the program still raise the error:

Traceback (most recent call last):
File "E:/Project/REDN-master/redn_trainer.py", line 195, in
train(**args_list[0])
File "E:/Project/REDN-master/redn_trainer.py", line 143, in train
_framework.model.infer({"token": ["he", "has", "combined", "intelligent", "history", "with", "moving", "prose", ",", "creating", "an", "informative", ",", "inspiring", "narrative", "telling", "the", "history", "of", "a", "great", "people", "."], "h": {"pos": [14, 15]}, "t": {"pos": [17, 18]}})
File "E:\Project\REDN-master\opennre\model\para.py", line 48, in infer
logits = self.softmax(logits)
File "C:\Users\HUI\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\HUI\Anaconda3\lib\site-packages\torch\nn\modules\activation.py", line 1044, in forward
return F.softmax(input, self.dim, _stacklevel=5)
File "C:\Users\HUI\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1442, in softmax
ret = input.softmax(dim)
AttributeError: 'tuple' object has no attribute 'softmax'

I am very grateful if anyone can help me to fix this.

Moreover, I use the _framework.model.infer instead of _framework.parallel_model.infer, but the original code train the model using _framework.parallel_model.load_state_dict, as _framework.parallel_model.infer is not implemented. Is that right?

这篇文章在测试的时候需要利用金标准的实体信息，而HBT这写关系抽取模型是预测实体和关系，这篇文章主要还是预测关系，假定实体给定，作者虽然在论文中提及了这个问题，但在结果比较中是不公平的，说自己的结果是SOTA也是不合理的。作者在文章中说自己的模型可以和NER结合，但代码中并没有体现这一点。

The task offical score is macro f1, (9+1)-way evaluation with directionality taken into account, also used in most of the papers, but the result reported in paper is micro f1?
thanks for your reply

When I am trying to train the model on GPU, I am getting RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50 error and it is impossible to train on CPU. I tried it on different GPUs on different computer still no luck and I am using them to train various pytorch models. Do you know why I am getting this error?

有处理过的数据最好，论文里提的有细节Supplementary Materials貌似也没有

Hello, I use the following command to run the code, but the feedback is as follows, please what SHOULD I do, thank you

python train.py train -s redn/output -f X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\training_configs\relation_extrac
tion\redn-webnlg.jsonnet

error info：
allennlp.common.checks.ConfigurationError: bert-pretrained not in acceptable choices for dataset_reader.token_indexers.tokens.type: ['single_id', 'characters', 'elmo_characters', 'spac
y', 'pretrained_transformer', 'pretrained_transformer_mismatched']. You should either use the --include-package flag to make sure the correct module is loaded, or use a fully qualified
class name in your config file like {"model": "my_module.models.MyModel"} to have it imported automatically.

my config file：
redn-webnlg.jsonnet
{
"dataset_reader": {
"type": "relation_extraction_reader",
"lazy":false,
"tokenizer":{
"type":"pretrained_transformer",
"model_name":"X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\data\uncased_L-12_H-768_A-12",
"do_lowercase":true
},
"token_indexers": {
"tokens": {
"type": "bert-pretrained",
"pretrained_model":"X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\data\uncased_L-12_H-768_A-12",
"do_lowercase": true,
"use_starting_offsets": true
}
}
},
"train_data_path": "X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\data\webnlg_release_v2_train.json",
"validation_data_path": "X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\data\webnlg_release_v2_dev.json",
"test_data_path": "X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\data\webnlg_release_v2_test.json",
"evaluate_on_test": true,
"model": {
"type": "redn",
"model_path":"X:\python\REDN-allennlp-based\REDN-allennlp-based\redn\data\uncased_L-12_H-768_A-12"
},
"iterator": {
"type": "bucket",
"sorting_keys": [["tokens","num_tokens"]],
"batch_size": 64
},
"trainer": {
"num_epochs": 50,
"patience": 0,
"cuda_device": 5,
"validation_metric": "+micro_f1",
"num_serialized_models_to_keep":3,
"optimizer": {
"type": "adam",
"lr":3e-5,
"weight_decay":1e-5
}

}
}

我看新分支里面的，一个instance一个关系，如何做到识别一个instance里面有多个关系呢？
如果一个instance里面，有多个不同实体对，是相同关系的如何构建输入的instance呢

I trained the model for nyt10 dataset for 10 epochs and accuracy exceeded 100% and I got 103%. Can you elaborate me on that?

Thanks

In OpenNRE nyt10 files are txt files so I'm trying to find json files but so far no luck for the original ones however I found pre-processed nyt10 .json files but when I ran the code no matter what the data is I am getting that error.

==========nyt10 None============
@@@@@@@@@@@ args @@@@@@@@@@@
{'metric': 'micro_f1', 'cuda_device': 3, 'seed': 31415926535897932, 'opt': 'adam', 'use_cls': True, 'subject_1': False, 'large_bert': False, 'continue_train': False, 'eval': False, 'add_subject_loss': False, 'weight_decay': 1e-05, 'lr': 5e-05, 'max_epoch': 100, 'sort': False, 'softmax': False, 'num_workers': 2, 'dataset_name': 'nyt10', 'batch_size': 20}
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Traceback (most recent call last):
  File "redn_trainer.py", line 205, in <module>
    train(**args_list[0])
  File "redn_trainer.py", line 92, in train
    train_loader, val_loader, test_loader = get_dataset(_model)
  File "redn_trainer.py", line 62, in get_dataset
    sort=sort), dataset_file))
  File "redn_trainer.py", line 62, in <lambda>
    sort=sort), dataset_file))
  File "/home/karaz/Desktop/Relation_Extraction/Relation_Extraction_English/REDN/opennre/framework/data_loader.py", line 37, in __init__
    self.load(sort, sort_reverse)
  File "/home/karaz/Desktop/Relation_Extraction/Relation_Extraction_English/REDN/opennre/framework/data_loader.py", line 48, in load
    _d["token"] = _d["text"].split(" ")
TypeError: list indices must be integers or slices, not str

In the master branch, the flag 'multi_label' is true on semeval. What's more, the calculation of f1 seems to ignore the case where the pred is positive, but the label is negative. I tried to solve these problems, and finally found that f1 could not reach the value in the paper.

I have red your code about evaluation. When compute f1 score, you matmul the pred_result and gold matrix mask, then compute the mean for every relation. I can not see how you extract the entity, and may I understand this as relation classification but not relation extraction?

Hi,
I was trying to run this model on the WebNLG data, but receive a JSONdecodeError. Looking at the opennre/framework/dataloader.py in opennre, it seems that it only decodes the json format used for the other benchmarks in OpenNRE.
So I'm wondering how you managed to train and test WebNLG, did you change the json format?

I've run - python redn_trainer.py semeval t
and is getting error No module named 'transformers.models'

transformers==3.4.0
appreciate your advise.

From the original paper, the NYT and WebNLG used by CasRel (HBT) separately contain 56195 + 5000 + 5000 sentences and 5019 + 500 + 703 sentences，which are inconsistent with the ones (69710, 31511) mentioned in your paper. It is unfair to compare with the results copied from the paper when using different datasets because the improvements may come from more training data. So, I recommend that reproduce HBT results on your datasets or use the datasets from HBT, and then give a more persuasive result.

Hi there!

I'm training on the NYT-10 dataset using the default hyperparameters from the repository, without changing anything but cuda_device, setting num_workers to 0 and batch_size to fit in my GTX-1070.

Training starts with a good accuracy/f1 in the first epoch, but decreases at every following epoch. Should I change anything else so the training can work on this dataset?

Thanks!

2点质疑：

1. 实体利用gold：不是联合抽取，结果不具备可比较性！
   2.实体pair也是直接根据gold给出：模型预测的实体pair也是利用gold，即预测的实体pair之间是一定有关系的，没有实体pair之间的冗余（负样本）问题？如下图所示，所以SOTA结果不可信！
   

gold_count = len(gold_label)
for idx, s in enumerate(res):
    if idx in gold_label and s == 1:
        self.res[CORRECT] += 1
        self.triple_count_res[triple_count][CORRECT] += 1 if idx != self.na_id else 0
        self.without_na_res[CORRECT] += 1 if idx != self.na_id else 0
        self.na_res[CORRECT] += 1 if idx == self.na_id else 0
        self.normal_res[CORRECT] += 1 if is_normal_data and idx != self.na_id else 0
        self.multi_label_res[CORRECT] += 1 if is_multi_label_data and idx != self.na_id else 0
        self.over_lapping_res[CORRECT] += 1 if is_over_lapping_data and idx != self.na_id else 0
        gold_count -= 1
    if idx in gold_label:
        self.res[GOLD_POSITIVE] += 1
        self.triple_count_res[triple_count][GOLD_POSITIVE] += 1 if idx != self.na_id else 0
        self.without_na_res[GOLD_POSITIVE] += 1 if idx != self.na_id else 0
        self.na_res[GOLD_POSITIVE] += 1 if idx == self.na_id else 0
        self.normal_res[GOLD_POSITIVE] += 1 if is_normal_data and idx != self.na_id else 0
        self.multi_label_res[GOLD_POSITIVE] += 1 if is_multi_label_data and idx != self.na_id else 0
        self.over_lapping_res[GOLD_POSITIVE] += 1 if is_over_lapping_data and idx != self.na_id else 0
    if s == 1:
        self.res[PRED_POSITIVE] += 1
        self.triple_count_res[triple_count][PRED_POSITIVE] += 1 if idx != self.na_id else 0
        self.without_na_res[PRED_POSITIVE] += 1 if idx != self.na_id else 0
        self.na_res[PRED_POSITIVE] += 1 if idx == self.na_id else 0
        self.normal_res[PRED_POSITIVE] += 1 if is_normal_data and idx != self.na_id else 0
        self.multi_label_res[PRED_POSITIVE] += 1 if is_multi_label_data and idx != self.na_id else 0
        self.over_lapping_res[PRED_POSITIVE] += 1 if is_over_lapping_data and idx != self.na_id else 0
self.res[TOTAL] += 1   # shouldn't here to add "gold_count" instead of 1?
self.triple_count_res[triple_count][TOTAL] += 1 if ep[2] != self.na_id else 0
self.without_na_res[TOTAL] += 1 if ep[2] != self.na_id else 0
self.na_res[TOTAL] += 1 if ep[2] == self.na_id else 0
self.normal_res[TOTAL] += 1 if is_normal_data and ep[2] != self.na_id else 0
self.multi_label_res[TOTAL] += 1 if is_multi_label_data and ep[2] != self.na_id else 0
self.over_lapping_res[TOTAL] += 1 if is_over_lapping_data and ep[2] != self.na_id else 0

In your f1_metric.py, when calculating the score, as self.res[TOTAL] += 1 instead of gold_count, it is possible that your accuracy exceeded 100% (if gold_count >= 2). I wonder if my understanding is wrong?

Where can I get the BERT pre-trained model that you used as you stated in your paper. I think you are referring to standard BERT trained by google research but I want to make sure. Also you haven't talked about inference in README, after training how can I do inference? Also where is that supplementary materials that you talked about in your thesis.

Thanks in advance

自定义数据集格式和semeval完全一样，只是关系的种类多一些。请指教是什么原因？

ValueError: Caught ValueError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/qwh/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/qwh/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/qwh/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/qwh/桌面/REDN/opennre/framework/data_loader.py", line 143, in getitem
seq_len)
File "/home/qwh/桌面/REDN/opennre/framework/data_loader.py", line 155, in merge_ont_hot_labels
em = SentenceREDataset.get_entity_mask(pos_head, pos_tail, new_index, seq_len)
File "/home/qwh/桌面/REDN/opennre/framework/data_loader.py", line 166, in get_entity_mask
pos_tail = [new_index.index(i) for i in pos_tail]
File "/home/qwh/桌面/REDN/opennre/framework/data_loader.py", line 166, in
pos_tail = [new_index.index(i) for i in pos_tail]
ValueError: 7 is not in list

4%|▍ | 4/104 [00:00<00:07, 12.73it/s]

When I am trying to load trained model with model.load_state_dict(torch.load("ckpt") I am getting this RuntimeError: storage has wrong size: expected -4883207186230854459 got 768 error. When I searched it says it might be related with workers overwriting on to the same file so how can I resolve this?

Thanks

This paper uses gold entity information rather than the predicted entities to evaluate the model.
It is inappropriate to say that you achieved the best results on NYT and WebNLG, since previous works (such as HBT) did not use the gold entity information when testing.

@@@@@@@@@@@ args @@@@@@@@@@@
{'metric': 'micro_f1', 'cuda_device': 3, 'seed': 31415926535897932, 'opt': 'adam', 'use_cls': True, 'subject_1': False, 'large_bert': False, 'continue_train': False, 'eval': False, 'add_subject_loss': False, 'weight_decay': 1e-05, 'lr': 5e-05, 'max_epoch': 100, 'sort': False, 'softmax': False, 'num_workers': 2, 'dataset_name': 'nyt10', 'batch_size': 20}
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
=== Epoch 0 train ===
  0%|                                                                                                                                                                              | 0/2840 [00:00<?, ?it/s]Traceback (most recent call last):
  File "redn_trainer.py", line 205, in <module>
    train(**args_list[0])
  File "redn_trainer.py", line 117, in train
    _framework.train_model(metric=metric)
  File "/home/turkai/kemal/REDN/opennre/framework/sentence_re.py", line 92, in train_model
    for iter, data in enumerate(t):
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/tqdm/std.py", line 1081, in __iter__
    for obj in iterable:
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
    return self._process_data(data)
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
    data.reraise()
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/torch/_utils.py", line 385, in reraise
    raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/turkai/anaconda3/envs/redn/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/turkai/kemal/REDN/opennre/framework/data_loader.py", line 143, in __getitem__
    seq_len)
  File "/home/turkai/kemal/REDN/opennre/framework/data_loader.py", line 155, in merge_ont_hot_labels
    em = SentenceREDataset.get_entity_mask(pos_head, pos_tail, new_index, seq_len)
  File "/home/turkai/kemal/REDN/opennre/framework/data_loader.py", line 164, in get_entity_mask
    pos_head = [new_index.index(i) for i in pos_head]
  File "/home/turkai/kemal/REDN/opennre/framework/data_loader.py", line 164, in <listcomp>
    pos_head = [new_index.index(i) for i in pos_head]
ValueError: 186 is not in list

I haven't changed anything but I am getting this error. What might be the reason? Alos tried with 0 num_workers, no luck so far.

Thanks

No module named "RelationExtractionWithNERReader" and "NERReader", could you provide the complete code?

I am getting a warning "Token indices sequence length is longer than the specified maximum sequence length for this model (730 > 512). Running this sequence through the model will result in indexing errors" will that cause a problem. I couldn't find a truncation operation or max_length used in BERTHiddenStateEncoder and I know BERT model is limited to 512 tokens so will that cause a decrease in performance and stuff?

Thanks