plkmo / bert-relation-extraction Goto Github PK

Hey plkmo,

I was wondering if the main_task.py could be used to train on a completely new set of relationships different from the semeval dataset. Could you please let me know the changes that I need to make in the code to train and predict on new data?

Thanks

Hi, @plkmo, first thank you for this repo, it is really nice.

I've trained a model using your script, this way:

python main_task.py \
    --model_no 1 \
    --model_size albert-large-v2 \
    --batch_size 16 \
    --lr 0.00004 \
    --train 1 \
    --infer 0

This generates a model in the data folder, called task_test_model_best_1.pth.tar.

Now I want to infer using this model that I previously trained, I tried doing this the following way:

!python main_task.py \
    --model_no 1 \
    --model_size "/path/to/model/task_test_model_best_1.pth.tar" \
    --train 0 \
    --infer 1

But when doing this, I'm getting the following error:

07/09/2020 10:28:10 PM [INFO]: Loading tokenizer and model...
07/09/2020 10:28:10 PM [INFO]: PyTorch version 1.5.1+cu101 available.
2020-07-09 22:28:10.268412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
07/09/2020 10:28:11 PM [INFO]: TensorFlow version 2.2.0 available.
Traceback (most recent call last):
  File "main_task.py", line 53, in <module>
    inferer = infer_from_trained(args, detect_entities=True)
  File "/content/BERT-Relation-Extraction/src/tasks/infer.py", line 68, in __init__
    task='classification', n_classes_=self.args.num_classes)
  File "/content/BERT-Relation-Extraction/src/model/ALBERT/modeling_utils.py", line 389, in from_pretrained
    **kwargs,
  File "/content/BERT-Relation-Extraction/src/model/ALBERT/configuration_utils.py", line 176, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/content/BERT-Relation-Extraction/src/model/ALBERT/configuration_utils.py", line 226, in get_config_dict
    config_dict = cls._dict_from_json_file(resolved_config_file)
  File "/content/BERT-Relation-Extraction/src/model/ALBERT/configuration_utils.py", line 315, in _dict_from_json_file
    text = reader.read()
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Am I doing this wrong? Could you please correct me if I'm using this script the wrong way?

In main_pretaining.py, line 21:
parser.add_argument("--pretrain_data", type=str, default="./data/cnn.txt", \ help="pre-training data .txt file path")

I don't find the cnn.txt. So where can downnload it? Thanks.
https://cs.nyu.edu/~kcho/DMQA/ Did you download it here？

Hi, plkmo !
I have managed to run the main_task script using follow command:
python main_task.py --train_data ./data/finetune/SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT --test_data ./data/finetune/SemEval2010_task8_all_data/SemEval2010_task8_testing_keys/TEST_FILE_FULL.TXT --use_pretrained_blanks 0 --model_no 0
but always got zero f1 in about 20 ecpos, you can see the detail in following:

Epoch finished, took 29.90 seconds. Losses at Epoch 20: 2.6830234 Train accuracy at Epoch 20: 0.1756250 Test f1 at Epoch 20: 0.0000000

HI, I have problem when infering from code

from src.tasks.infer import infer_from_trained

inferer = infer_from_trained(args, detect_entities=False)
test = "The surprise [E1]visit[/E1] caused a [E2]frenzy[/E2] on the already chaotic trading floor."
inferer.infer_sentence(test, detect_entities=False)

It throws error that args is not defined, can u help me with which values to fill args. If I put None it needs the args.pkl, but I cannot find the pickle after the training for semeval task.

I'm trying to run the main_task.py but its showing an error, Can someone explain how and in which order to execute the files?

Traceback (most recent call last):
File "d:\6th Sem\NLP\Project\BERT-Relation-Extraction-master\main_task.py", line 50, in
net = train_and_fit(args)
File "d:\6th Sem\NLP\Project\BERT-Relation-Extraction-master\src\tasks\trainer.py", line 33, in train_and_fit
train_loader, test_loader, train_len, test_len = load_dataloaders(args)
File "d:\6th Sem\NLP\Project\BERT-Relation-Extraction-master\src\tasks\preprocessing_funcs.py", line 319, in load_dataloaders
save_as_pickle("%s_tokenizer.pkl" % model_name, tokenizer)
File "d:\6th Sem\NLP\Project\BERT-Relation-Extraction-master\src\misc.py", line 22, in save_as_pickle
with open(completeName, 'wb') as output:
FileNotFoundError: [Errno 2] No such file or directory: './data/BERT_tokenizer.pkl'

Hi,
I wanted to know whether this implementation gives around 70.1 F1 for BERT-EM model as reported in the paper on TACRED dataset.

Thanks and regards,

Please help, how can we get all the P.R.F1 by modify the code ? (/▽＼)

Hello,

if we change line 420 in /src/preprocessing_funcs.py from
nlp = spacy.load("en_core_web_lg")
to
nlp = en_core_web_sm.load()
and add
import en_core_web_sm
at the beginning of the file, the error
OSError: [E050] Can't find model 'en_core_web_lg'. It doesn't seem to be a Python package or a valid path to a data directory.
is fixed.
(As described here explosion/spaCy#4577)

Hi, you mention the pretrained Bert model as your requirement. But you do your own pretraining right?

The downloadable pretrained model is done with corpus that does not come with labels for relation extraction.

It sounds like you do your pretraining with text automatically given labels for entity and relation by spacy which may not be 100% correct. Am I correct in this?

For the training (I suppose this is fine-tuning), you use the Semeval dataset whose labels for entities and relations are supposed to be manually checked and 100% correct.

I suppose you have tried with the downloadable pretrained model that does not get any label for entities and relations. How much worse the performance would be when you do this?

Thanks.

To use BioBERT(biobert_v1.1_pubmed), download & unzip the contents to ./additional_models folder.

However, this link is not useful now.

Thank you so much for the clear code.

Does the classification model use any development set as the paper?! How could it be added to the code?

Hi, I find your accuracy is calculated on training data, but generally we evaluate the model on test data. I get the accuracy from test data and the result is not that good, is there anything wrong?

Currently I am pre-training the bert model for MTB but how can I use that pre-trained model for fine-tuning for SemEval2010 Task 8? I saw that you gave information about how to fine-tune the actual bert model for SemEval2010 Task 8 however I couldn't find any information about how to fine tuned pre-trained MTB model? In addition to that after fine-tuning on SemEval2010 Task 8 should I do NER first or the model does that itself?

Thanks

I want to run only the "Fine-tuning on SemEval2010 Task 8" section (main_task.py file). However, I get the following error:
FileNotFoundError: [Errno 2] No such file or directory: './data/BERT_tokenizer.pkl'

Can anyone tell me where I can get this file?

I clone github code and i was trying to execute main_pretraining.py with all dependecies and with default arguments. I am facing following issues during training process:
File "main_pretraining.py", line 35, in
output = train_and_fit(args)
File "/home/eswar/yamuna_projects/BERT-Relation-Extraction/src/trainer.py", line 79, in train_and_fit
for i, data in enumerate(train_loader, 0):
File "/home/eswar/yamuna_projects/BERT-Relation-Extraction/src/preprocessing_funcs.py", line 246, in getitem
size=min((self.batch_size - 1), len(pool)), replace=False)
File "mtrand.pyx", line 1126, in mtrand.RandomState.choice
ValueError: a must be non-empty

Hi,
Thanks for sharing the implementation of "Matching the Blank" paper. I just wanted to know what is the benefit of changing gradient_acc_steps parameter? Does it make training/convergence faster?

Thanks and regards

Hi, I just trained my model locally and checked the results of my trained models against the ones on the README. I found that they are different. I believe this is due to the embeddings of the previously mentioned tokens change every time the model is instantiated. For instance, trying with the same phrase, if I instantiated the model and predicted, the output would be different from the next time I instantiated and predicted the same phrase.

I believe in the __init__ method of the infer_from_trained class, with the method resize_token_embeddings()at line 83 of the infer.py file, the embeddings are being extended to have the 4 extra tokens, but the embeddings are being initialized randomly and this causes the results to vary.

Am I understanding it correctly? Or am I mistaken? Any help would be appreciated.

I do not find any code about MTB loss, such as dot product performed before a binary classifier.

Hi @plkmo !

Great work!

Do you have the loss curve available from your trainings (pre-training and semeval training) so we can check if our experiments are matching yours?

Thanks!

After training on my own data and then going to infer, if I type in one of the example sentences given, the expected result comes back. When I enter a typical sentence from my training data, I get the following error:

File "main_task.py", line 63, in

inferer.infer_sentence(sent, detect_entities=False)
File "/home/gregory.werner/BERT-Relation-Extraction/src/tasks/infer.py", line 222, in infer_sentence
return self.infer_one_sentence(sentence)
File "/home/gregory.werner/BERT-Relation-Extraction/src/tasks/infer.py", line 193, in infer_one_sentence
e1_e2_start = self.get_e1e2_start(tokenized); #print(e1_e2_start)
File "/home/gregory.werner/BERT-Relation-Extraction/src/tasks/infer.py", line 186, in get_e1e2_start
e1_e2_start = ([i for i, e in enumerate(x) if e == self.e1_id][0],
IndexError: list index out of range

Hi
How could I access a relation representation based on BERT(EM) and BERT(EM+MTB) models?

Hi.
Thank you for the implementation. It is great to learn from your code.
I have observed a weird thing wrt FewRel. The answer in the training set is always 4, as I added in evaluate(),

                if meta_labels[-1].item() != 4:
                    print(closest_idx, 'vs', meta_labels[-1].item())

Nothing was printed. The predicted result closest_idx varies, so there is no problem with the model. The implementation looks correct as well. It might be caused by the implementation of meta_labels, for example,

for sample_idx, r in enumerate(sampled_relation):
...
        meta_input = meta_train_input + [torch.LongTensor(meta_test_input)]
        meta_labels = meta_train_labels + [meta_test_labels]
        print('meta_labels =', meta_labels)   # Added this line

Also, meta_labels[-1] was obtained from,

        target_idx = self.N - 1
...
        meta_test_labels = [target_idx]
...
        meta_labels = meta_train_labels + [meta_test_labels]

It is fixed to self.N-1, thus 4 in 5-way-1-shot.

meta_labels = [[0], [1], [2], [3], [4], [4]]
4 vs 4
meta_labels = [[0], [1], [2], [3], [4], [4]]
1 vs 4
meta_labels = [[0], [1], [2], [3], [4], [4]]
4 vs 4
meta_labels = [[0], [1], [2], [3], [4], [4]]
1 vs 4
meta_labels = [[0], [1], [2], [3], [4], [4]]
4 vs 4

Hi,

I would like to ask you to publish your trained model when it is only trained on TACRED ( without the matching the blanks training on Wikipedia).
I want to see the model prediction per instance when the model is trained based on TACRED only.
Alternatively, if you could share with me the command to train your model only based on TACRED.

All the Best,

Hi
Thanks for the brilliant implementation.
Do you or anyone from the community have trained the model and can give access for downloading for everyone usage?
I don't want to train an already trained model, it's not healthy for the environment!
Thanks.

       Please help,when I run the main_task.py,it occurred the error: 
       IndexError: ('list index out of range', 'occurred at index 47') (main_pretraining.py)

I tried all I can but it just didn't work.

        please help , xiexie

We use your albert pretrained checkpoint files, and the finetune. The f1 0.5978749489170413. Why the value is so low. We only use the default value of the args.

Hi,

I was trying to fine tune the model on SemEval2010 Task 8 dataset on google colab. I tried the following code

!rm -rf ./data
!mkdir ./data
!unzip <path1>/SemEval2010_task8_all_data.zip -d ./data/

!python <path2>/BERT-Relation-Extraction/main_task.py  \
--train_data ./data/SemEval2010_task8_all_data/SemEval2010_task8_training/TRAIN_FILE.TXT \
--test_data ./data/SemEval2010_task8_all_data/SemEval2010_task8_testing/TEST_FILE.txt \
--num_classes 9  \
--batch_size 100 \
--num_epochs 3 \
--lr 0.0001 \
--model_no 0 \
--model_size bert-base-uncased \
--train 1

But this throws an error, saying:

Traceback (most recent call last):
  File "<whatever>/BERT-Relation-Extraction/main_task.py", line 50, in <module>
    net = train_and_fit(args)
  File "<whatever>/BERT-Relation-Extraction/src/tasks/trainer.py", line 33, in train_and_fit
    train_loader, test_loader, train_len, test_len = load_dataloaders(args)
  File "<whatever>/BERT-Relation-Extraction/src/tasks/preprocessing_funcs.py", line 336, in load_dataloaders
    df_train, df_test, rm = preprocess_semeval2010_8(args)
  File "<whatever>/BERT-Relation-Extraction/src/tasks/preprocessing_funcs.py", line 67, in preprocess_semeval2010_8
    sents, relations, comments, blanks = process_text(text, 'test')
  File "<whatever>/BERT-Relation-Extraction/src/tasks/preprocessing_funcs.py", line 39, in process_text
    assert re.match("^Comment", comment)
AssertionError

What am I missing here?

self.rm = load_pickle("relations.pkl"), i dont know where to find it...

您好，我在SemEval2010_task8数据集上运行您的代码时，总出现过拟合的情况，test的f1一直在0.45左右，请问这种情况该如何处理？

Hi

I was trying to run your code on another task. I have processed the data to be similar to semeval data. I have also amended the number of relations and the 2 files for evaluation with my relation types. However, I keep getting this error, any idea why?

[Epoch: 1, 3216/ 32185 points] total loss, accuracy per batch: 0.703, 0.733
[Epoch: 1, 6432/ 32185 points] total loss, accuracy per batch: 0.610, 0.774
[Epoch: 1, 9648/ 32185 points] total loss, accuracy per batch: 0.567, 0.794
[Epoch: 1, 12864/ 32185 points] total loss, accuracy per batch: 0.577, 0.780
[Epoch: 1, 16080/ 32185 points] total loss, accuracy per batch: 0.561, 0.785
[Epoch: 1, 19296/ 32185 points] total loss, accuracy per batch: 0.536, 0.793
[Epoch: 1, 22512/ 32185 points] total loss, accuracy per batch: 0.545, 0.799
[Epoch: 1, 25728/ 32185 points] total loss, accuracy per batch: 0.540, 0.792
[Epoch: 1, 28944/ 32185 points] total loss, accuracy per batch: 0.522, 0.799
[Epoch: 1, 32160/ 32185 points] total loss, accuracy per batch: 0.489, 0.810
06/21/2020 12:24:46 AM [INFO]: Evaluating test samples...
0%| | 1/1644 [00:15<7:00:27, 15.35s/it]
Traceback (most recent call last):
File "main_task.py", line 48, in
net = train_and_fit(args)
File "/Volumes/ghada/nlp/BERT-Relation-Extraction-master/src/tasks/trainer.py", line 159, in train_and_fit
results = evaluate_results(net, test_loader, pad_id, cuda)
File "/Volumes/ghada/nlp/BERT-Relation-Extraction-master/src/tasks/train_funcs.py", line 93, in evaluate_results
e1_e2_start=e1_e2_start)
File "/Users/ghada/opt/anaconda3/envs/relationBert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/Volumes/ghada/nlp/BERT-Relation-Extraction-master/src/model/BERT/modeling_bert.py", line 734, in forward
embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
File "/Users/ghada/opt/anaconda3/envs/relationBert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/Volumes/ghada/nlp/BERT-Relation-Extraction-master/src/model/BERT/modeling_bert.py", line 177, in forward
position_embeddings = self.position_embeddings(position_ids)
File "/Users/ghada/opt/anaconda3/envs/relationBert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/Users/ghada/opt/anaconda3/envs/relationBert/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/Users/ghada/opt/anaconda3/envs/relationBert/lib/python3.6/site-packages/torch/nn/functional.py", line 1724, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

Can you pls share your model file for bertEM + MTB?

In the original paper, different input/output types are evaluated (please look at Table 1), it seems that current implementation only support [CLS] output type.

Does it correct or I missed something?

Thanks.

Hi,

Thanks for providing the biobert code.

To use BioBERT(biobert_v1.1_pubmed), download & unzip the contents to ./additional_models folder.

However, this link is not useful now.

Where could I get the contents?

The original function has a problem. If "E1" is "the apple" and "E2" is "the banana", "E2" cannot be annotated because of "continue" in "E1" annotation section.
For example, how about "I prefer the banana to the apple."?
E1 == apple ,E2 == banana here.
The word "the" in E2"the banana" wouldn't be recognized instead of detection of E1"the apple".
That's why I changed the function not to avoids the problem. New function is below.
My function may be a bad code and increase execution time, but it can improve E2 detection.

def annotate_sent(self, sent_nlp, e1, e2):
        annotated = ''
        e1start, e1end, e2start, e2end = 0, 0, 0, 0
        for i,token in enumerate(sent_nlp):
            if not isinstance(e1, list):
                if (token.text == e1.text) and (e1start == 0) and (e1end == 0):
                    e1spos=i
                    e1epos=i
                    e1start, e1end = 1, 1
            elif len(e1) == 1:
                if (token.text == e1[0].text) and (e1start == 0) and (e1end == 0):
                    e1spos=i
                    e1epos=i
                    e1start, e1end = 1, 1
            else:
                if (token.text == e1[0].text) and (e1start == 0):
                    e1spos=i
                    e1start += 1
                elif (token.text not in [i.text for i in e1]) and (e1start == 1) and (e1end == 0):
                    e1start = 0
                elif (token.text == e1[-1].text) and (e1end == 0) and (e1start == 1):
                    e1epos=i
                    e1end += 1
           
            if not isinstance(e2, list):
                if (token.text == e2.text) and (e2start == 0) and (e2end == 0):
                    e2spos=i
                    e2epos=i
                    e2start, e2end = 1, 1
                    continue
            elif len(e2) == 1:
                if (token.text == e2[0].text) and (e2start == 0) and (e2end == 0):
                    e2spos=i
                    e2epos=i
                    e2start, e2end = 1, 1
                    continue
            else:
                if (token.text == e2[0].text) and (e2start == 0):
                    e2spos=i
                    e2start += 1
                    continue
                elif (token.text not in [i.text for i in e2]) and (e2start == 1) and (e2end == 0):
                    e2start = 0
                elif (token.text == e2[-1].text) and (e2end == 0) and (e2start == 1):
                    e2epos=i
                    e2end += 1
                    continue
        for i,token in enumerate(sent_nlp):
            if i == e1spos:
                annotated += ' [E1]'
            if i == e2spos:
                annotated += ' [E2]' 
            annotated += ' ' + token.text + ' '
            if i == e1epos:
                annotated += '[/E1] '
            if i == e2epos:
                annotated += '[/E2] '
            
        annotated = annotated.strip()
        annotated = re.sub(' +', ' ', annotated)
        return annotated

@plkmo I am very impressed with the clean and organized code! It really helps me a lot!

I have a problem that when I try to run on model size =='bert-large-uncased'. There is a mismatch problem:
Traceback (most recent call last):
File "D:\BERT\BERT-Relation-Extraction-master\BERTtest.py", line 316, in
e1_e2_start=e1_e2_start)
File "C:\Users\qingd\anaconda3\envs\P37\lib\site-packages\torch\nn\modules\module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "D:\BERT\BERT-Relation-Extraction-master1\BERT-Relation-Extraction-master\src\model\BERT\modeling_bert.py", line 754, in forward
classification_logits = self.classification_layer(v1v2)
File "C:\Users\qingd\anaconda3\envs\P37\lib\site-packages\torch\nn\modules\module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "C:\Users\qingd\anaconda3\envs\P37\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "C:\Users\qingd\anaconda3\envs\P37\lib\site-packages\torch\nn\functional.py", line 1369, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [32 x 2048], m2: [1536 x 19] at C:/w/1/s/tmp_conda_3.7_055457/conda/conda-bld/pytorch_1565416617654/work/aten/src\THC/generic/THCTensorMathBlas.cu:273

Could you please give me some clues about how to deal with this problem!

Thanks a lot!

Hi

How do I use this for few-shot classification task. Pls advise.

HI,

Thank you for your code and for adding BioBert.
Tried to use your code with SciBert. For this, I created a code 3 to specify the model type in the input (just as you introduced 2 for BioBert), and added the necessary specification everywhere (following BioBert example).

When training, there is the following error:

12/18/2020 03:02:57 PM [INFO]: Starting training process...

Traceback (most recent call last):
File "main_task.py", line 52, in
net = train_and_fit(args)
File "/mnt/irisgpfs/users/mbiryukov/BertRel-UPD/BERT-Relation-Extraction/src/tasks/trainer.py", line 161, in train_and_fit
e1_e2_start=e1_e2_start)
File "/opt/apps/resif/data/production/v1.2-20191021/default/software/devel/PyTorch/1.2.0-fosscuda-2019a-Python-3.7.2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/mnt/irisgpfs/users/mbiryukov/BertRel-UPD/BERT-Relation-Extraction/src/model/BERT/modeling_bert.py", line 759, in forward
classification_logits = self.classification_layer(v1v2)
File "/opt/apps/resif/data/production/v1.2-20191021/default/software/devel/PyTorch/1.2.0-fosscuda-2019a-Python-3.7.2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 591, in getattr
type(self).name, name))
AttributeError: 'BertModel' object has no attribute 'classification_layer'

Could you please tell what should be modified (m.b. in model config or train?) to get it work?

Thank you!

Hi! Thanks so much for providing this implementation of the MTB training strategy.

I noticed that in the paper the authors use noise contrastive estimation in the training scheme, whereas in this implementation there seems to be an internal batching flag within the dataloader in preprocessing_funcs, which enables noise contrastive estimation.

See here:

Is there a reason for this decision? Has anyone tried using standard batching rather than NCE?

I'll also try standard batching myself and update this thread if I have any meaningful results.

@plkmo In main_task.py, the comments suggest you can fine-tune on either SemEval or FewRel. However, the README file gives results that are not trained on an FewRel data. It appears to me that only inference is taking place in regards to FewRel. Is this so, or is there a way to fine-tune on my own FewRel data (and save a resultant model files)?

A few more words in the FewRel Task README.md file would also be helpful.

What is the purpose of adding config=>>>fine_tuning_task=semeval
If I use it on Chinese what should I be aware of?
thanks

I take num-classes to be the distinct number of relation classes which can be annotated against and predicted. With this statement, I struggle with the two following points:

1. If we are telling the training code how many classes there are, shouldn't we be telling it which classes we are using?
2. Shouldn't we be able to look at the training data and pick out how many classes there are and what they are? This would seem the most sensible solution.

Hi,

In the paper, the author tested 6 architectures:
Figure 3: Variants of architectures for extracting relation representations from deep Transformers network. Figure (a) depicts a model with STANDARD input and [CLS] output, Figure (b) depicts a model with STANDARD input and MENTION POOLING output and Figure (c) depicts a model with POSITIONAL EMBEDDINGS input and MENTION POOLING output. Figures (d), (e), and (f) use ENTITY MARKERS input while using [CLS], MENTION POOLING, and ENTITY START output, respectively.

For BERT-EM model in your code, can I check if you are using (f) the model using the ENTITY MARKERS input representation and ENTITY START output representation?

Are you able to share which section of the code did you adapt this architecture?

Thanks!