About 'X' label about bert-ner HOT 22 CLOSED

Label 'X' is not considered for f1 metrics

from bert-ner.

But did you use in train?

from bert-ner.

For training I used "X".
While Inference and f1 metrics only the first output label of token is considered . same as in BERT paper

from bert-ner.

I think add more label in the conll2003 NER standard dataset make it not very comparable for previous works. Could you remove 'X' label during training and get a similar result?

from bert-ner.

If you remove "X" while training or replace "X" with "O" , model performance will drop to ~89 f1 score

from bert-ner.

So this is my opinion, use ‘X’ label make it high F1-score, it's not fair. I get the similar result about 91.3 F1 score. And I think BERT origin paper is also remove 'X' label because they use document information to get high F1-score. In short, 'X' label don't have any signal.

from bert-ner.

91.3 without using "X" ??

from bert-ner.

Yes. I get the word piece output from BERT model, and then map the first token's vector so I get the same numbers vectors as the standard dataset. And then use a softmax matrix to get the final result. But I only use the BERTModel in pytorch_pretrained_bert.

from bert-ner.

For example
After extracting features for the sentence below
Jim Hen ##son was a puppet ##eer
You're giving only [Jim , Hen , was , a, puppet] hidden states to linear classification layer ?

from bert-ner.

Yes. Because I think the fine tune bert could learn this pattern.

from bert-ner.

I will try this way and let you know

from bert-ner.

@kugwzk
Can you share your code ?
How are you handling padding after extracting first sub-token hidden states from BERT ?

from bert-ner.

I just record the origin word position use a dict in python. For example: [Jim Hen ##son was a puppet ##eer] for [0,1,3,4,5], so I padding the origin word sequence again in the classifier layer. It may be slowly :).

from bert-ner.

i think we can add mask to X label when training

from bert-ner.

I agree with @kugwzk on "misusage of X label". @tkukurin In the latest version, are you still using X during training (or evaluation) or have you already removed it as suggested ?

from bert-ner.

@ereday
Latest version still use X , I have code without usingX label , I need to clean the code a bit. I will try to push code by monday

from bert-ner.

Hi @kamalkraj ,

I am Nic from NVIDIA, thanks for your contribution on this project!
I tried to replace [CLS], X with O directly, and removed [SEP].
I think you supposed to release new code today, have you finished?
Thanks.

from bert-ner.

@toxic2m
Check out branch experiment

from bert-ner.

Hi @kamalkraj ,

Actually, I already done this part locally, and I suggest you to map [CLS] and [SEP] to O directly.
Then your FC layer only output real number of classifications, can get better performance.
Thanks.

from bert-ner.

Is there anyone able to reproduce BERT_NER paper's results (92.4F1 for BERT Base) ?
The experiment on the master branch supports the result given in the BERT paper. But as @kugwzk mentioned the problem. Is there anyone able to reproduce the results without inferred X tags?

from bert-ner.

@sbmaruf The result of Conll03 NER reported in the BERT origin paper used document context, which is different from the standard sentence-based evaluation. You can see something about that in here: allenai/allennlp#2067 (comment)

from bert-ner.

@kugwzk thanks for your reply. but how to add document level context with NER?
any idea or code repo?

from bert-ner.