microsoft / mpnet Goto Github PK
View Code? Open in Web Editor NEWMPNet: Masked and Permuted Pre-training for Language Understanding https://arxiv.org/pdf/2004.09297.pdf
License: MIT License
MPNet: Masked and Permuted Pre-training for Language Understanding https://arxiv.org/pdf/2004.09297.pdf
License: MIT License
多谢多谢!
@tobyoup @StillKeepTry
I tried Training MPNet on SQUAD2 data below is the result I was getting on Evalutionset
I used this script
(exact, 50.07159100480081)
(f1, 50.07159100480081)
(total, 11873)
(HasAns_exact, 0.0)
(HasAns_f1, 0.0)
(HasAns_total, 5928)
(NoAns_exact, 100.0)
(NoAns_f1, 100.0)
(NoAns_total, 5945)
(best_exact, 50.07159100480081)
(best_exact_thresh, 0.0)
(best_f1, 50.07159100480081)
(best_f1_thresh, 0.0)
有谁知道?
When I tried to train MPNet on the dataset wikitext-103, I got this error
请问,论文中SQuAD2.0的dev结果是采用默认阈值0 (是否有答案),还是best f1和best em呢? 非常感谢~
恭喜MPNet被NIPS2020接收,在最新一版的MPNet论文中,请问GLUE的dev集合中RTE和MRPC以及STS-B这几个任务有没有用MNLI checkpoint热启动呢?感谢~
https://github.com/microsoft/DeepSpeed
MPNet suffer from slow training time, deepspeed could significantly reduce the time needed and transformers (hugginface) support it apparently, any guide/sample code from how to enable it for MPNet?
When running the Training script for SQUAD I was getting the below error.
Traceback (most recent call last):
File "/media/data2/anaconda/envs/mpnet/bin/fairseq-train", line 33, in <module>
sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-train')())
File "/media/data1/bhadresh/MPNet/MPNet/pretraining/fairseq_cli/train.py", line 370, in cli_main
main(args)
File "/media/data1/bhadresh/MPNet/MPNet/pretraining/fairseq_cli/train.py", line 47, in main
task = tasks.setup_task(args)
File "/media/data1/bhadresh/MPNet/MPNet/pretraining/fairseq/tasks/__init__.py", line 17, in setup_task
return TASK_REGISTRY[args.task].setup_task(args, **kwargs)
File "/media/data1/bhadresh/MPNet/MPNet/pretraining/fairseq/tasks/squad2.py", line 104, in setup_task
return cls(args, dictionary)
File "/media/data1/bhadresh/MPNet/MPNet/pretraining/fairseq/tasks/squad2.py", line 84, in __init__
self.tokenizer = SQuADTokenizer(args.bpe_vocab_file, dictionary)
File "/media/data1/bhadresh/MPNet/MPNet/pretraining/fairseq/tasks/squad2.py", line 42, in __init__
self.max_len_single_sentence = self.max_len - 2
File "/media/data2/anaconda/envs/mpnet/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1547, in max_len_single_sentence
raise ValueError(
ValueError: Setting 'max_len_single_sentence' is now deprecated. This value is automatically set up.
By commenting out line 42 and 43 in file
self.max_len_single_sentence = self.max_len - 2
self.max_len_sentences_pair = self.max_len - 3
It resolves but is it fine to do so?
When I run the script, I was getting less F1 score and Exact Match than mentioned in the Paper. I also created an issue for that
Is there results on GLUE for MPNet which is pre-trained on 16G corpora? Only results of models trained on 160G corpora. Thanks~
你好请问一下你们的结果有在https://rajpurkar.github.io/SQuAD-explorer/这个排行榜上吗?
l enforce it to cuda, but throw exception (max recursive)
Hello,
Thank you for releasing the codes for pretraining MPNet!
I am trying to continue training of the language model task on a custom dataset from the released checkpoint using the --restore-file
argument. However, I am not able to successfully load the checkpoint. It fails with the following error: MPNet/pretraining/fairseq/checkpoint_utils.py", line 307, in _upgrade_state _dict registry.set_defaults(state['args'], tasks.TASK_REGISTRY[state['args'].task]) KeyError: 'mixed_position_lm'
In case it helps, here is the details of the training command :
WARMUP_UPDATES=50000 # Warmup the learning rate over this many updates
PEAK_LR=0.0005 # Peak learning rate, adjust as needed
TOKENS_PER_SAMPLE=512 # Max sequence length
MAX_POSITIONS=512 # Num. positional embeddings (usually same as above)
MAX_SENTENCES=35 # Number of sequences per batch (batch size)
UPDATE_FREQ=16 # Increase the batch size 16x
DATA_DIR=data-bin
fairseq-train --fp16 $DATA_DIR \
--task masked_permutation_lm --criterion masked_permutation_cross_entropy \
--arch mpnet_base --sample-break-mode none --tokens-per-sample $TOKENS_PER_SAMPLE \
--optimizer adam --adam-betas '(0.9,0.98)' --adam-eps 1e-6 --clip-norm 0.0 \
--lr-scheduler polynomial_decay --lr $PEAK_LR --warmup-updates $WARMUP_UPDATES \
--total-num-update $TOTAL_UPDATES --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
--max-sentences $MAX_SENTENCES --update-freq $UPDATE_FREQ --skip-invalid-size-inputs-valid-test \
--max-update $TOTAL_UPDATES --log-format simple --log-interval 1 --input-mode 'mpnet'\
--restore-file mpnet.base/mpnet.pt --save-interval-updates 10 --ddp-backend no_c10d
I will appreciate insights on what to do to resolve this error. Thank you!
could you release the finetune performance with fewer epoch on glue, such as epoch 3 the same as bert or roberta?
For example, it could really make sense to adapt MPNet to preserve PLM but uses the approach of ELECTRA for MLM.
SpanBERT has some potential too (e.g on coreference resolution)
I believe this could really push the state of the art of accuracy on key tasks.
What do you think?
@StillKeepTry
@tan-xu
Moreover there are important low hanging fruits that have been consistently ignored by transformer researchers:
The activation function used should probably be https://github.com/digantamisra98/Mish
as it is the one that give the most accuracy gains in general. It can give 1% accuracy gains which is huge.
Secondly the optimizer you're using, Adam is flawed and you should use its rectified version:
https://github.com/LiyuanLucasLiu/RAdam
Moreover it can be optionally combined with a complementary optimizer:
https://github.com/michaelrzhang/lookahead
Moreover there are newer techniques for training that yield significant accuracy gains, such as:
https://github.com/Yonghongwei/Gradient-Centralization
And gradient normalization.
There is a library that integrate all those advances and more here:
https://github.com/lessw2020/Ranger21
Accuracy gains in NLP/NLU have reached a plateau. The reason is that researchers works far too much in isolation. They bring N new innovations per years but the number of researchers that attempt to use those innovations/optimization together can be counted on the fingers of one hand.
XLnet has been consistently ignored by researchers, you are the ones that saw the opportunity to combine the best of both worlds of BERT and XLnet. Why stop there?
As I said, both transformer/language model wise and activation function/optimizer wise there are a LOT of significant accuracy optimizations to integrate into the successor of MPNet.
Aggregating those optimizations could yield a revolutionary language model that would have 5-10% accuracy gains on average over existing SOTA. It would mark history.
No one will attempt to combine a wide range of those innovations, you are the only hope. I you do not do it, I'm afraid no one else will and NLU will stagnate for the decade to come.
Hi all on the MPNet research team,
I am in the process of converting the fairseq training code for MPNet into a training loop that is compatible with Huggingface. Although many of the convenience classes already exist in Huggingface (like MPNetForMaskedLM
), one thing that has become clear to us is that we will need to port over the collator function in MaskedDataset
(under tasks/masked_permutation_lm
).
In exploring how this collator works, I understand the logic as:
n
indices of the permuted sequence, where n
is the prediction size (i.e. seq_len
x 0.15
at default values)concat(seq, mask, mask)
and concat(positions, predict_positions, predict_positions)
Using this logic, we might expect the collator function to perform the below operation on some dummy input IDs:
src_tokens = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
# Once the collator permutes everything and we append the mask portions, we expect something like
new_ids = [ 20, 23, 30, 14, 15, 27, 28, 11, 12, 17, 18, 26, 29, 13, 10, 19, 21, 22, 16, 24, 25, <mask>, <corrupted>, <mask>, <mask>, <corrupted>, <mask>]
new_positions = [10, 13, 20, 4, 5, 17, 18, 1, 2, 7, 8, 16, 19, 3, 0, 9, 11, 12, 6, 14, 15, 6, 14, 15, 6, 14, 15]
However, after rereading the MPNet paper, especially section 2.2 and 2.3 with attention on Figure 2, it would SEEM that the output of the collator is incongruous with what is described in these sections.
Figure 2 points out that the content and query masks are built using a permuted sequence that looks like:
src_tokens = [x_1, x_2, x_3, x_4, x_5, x_6]
# Once permuted we get:
new_ids = [x_1, x_3, x_5, <mask>, <mask>, <mask>, x_4, x_6, x_2]
new_positions = [1, 3, 5, 4, 6, 2, 4, 6, 2]
In this example within the paper, we are masking the pred_len
tokens and then appending the content to the end for the content stream. However, the collator output KEEPS the token content in the main sequence, and then adds TWO batches of mask tokens to the end, which to me seems necessarily different than what's described in the paper. Referring back to our dummy example above, I can outline the discrepancies I'm seeing:
collator_ids = [ 20, 23, 30, 14, 15, 27, 28, 11, 12, 17, 18, 26, 29, 13, 10, 19, 21, 22, 16, 24, 25, <mask>, <corrupted>, <mask>, <mask>, <corrupted>, <mask>]
collator_positions = [10, 13, 20, 4, 5, 17, 18, 1, 2, 7, 8, 16, 19, 3, 0, 9, 11, 12, 6, 14, 15, 6, 14, 15, 6, 14, 15]
paper_ids = [ 20, 23, 30, 14, 15, 27, 28, 11, 12, 17, 18, 26, 29, 13, 10, 19, 21, 22, <mask>, <corrupted>, <mask>, 16, 24, 25]
paper_positions = [10, 13, 20, 4, 5, 17, 18, 1, 2, 7, 8, 16, 19, 3, 0, 9, 11, 12, 6, 14, 15, 6, 14, 15]
My question, then, is this: am I correct in understanding that the collator implementation is different than what's described in the paper? If so, why?
would you release the code about convert the mpnet to the transformers "microsoft/mpnet-base"?
I want to convert my pretrain one to the transformers format
Hi I would like access the released fairseq checkpoint model - to continue pretraining on it
https://modelrelease.blob.core.windows.net/pre-training/MPNet/mpnet.base.tar.gz
This URL is not accessible to the public
is it input mode use mlm
is like roberta, and use plm
is like xlnet?
and by the way, would you provide the script to convert the model format to the huggingface transformers ones?
Any one know where to get them?
Thank you and thank you.
是否采用[:128] + [382:]这种截断方式么?感谢~
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.