ymcui / chinese-electra Goto Github PK
View Code? Open in Web Editor NEWPre-trained Chinese ELECTRA（中文ELECTRA预训练模型）
Home Page: http://electra.hfl-rc.com
License: Apache License 2.0
chinese-electra's People
Contributors
Stargazers
Watchers
Forkers
jjwangnlp gokunwu abner-lp liuwq168 oswen kinghuin yyht aiainui qianrenjian luoyexuge xhyandwyy hommmm airaria toastynews jalused rosssong chenmoshushi leileixiao chenruiqing onlyonewater humdingers awoziji shadowkun duxiaochao yangbin1729 18106574249 richiesui superrichiesui dongdong199408 duyuankai1992 leyiwang markclemens301 maozida880 yaoleihxr ljt270864457 jinsongpan askintution shi-xin anatanick taomiao xuj laomagic fantasyyyy manhlab flysky2008 daishu7 xwixcn yz-liu hmy626 elliotthwang yukuotc githubmyk bluestinger1989 aloha0424 gdh756462786 sunnyhuma171 syzong ictwang brucekyle99 guojson xiaming9880 zongmingshu dx2048 frankswu tianxie1012 linhanxiao xrosliang littck yijiantx lcg22 yukunren cclauss wuhenbai acproject communicateconnectcreate charliezhugj azuk1 lhb2017 tubbz-alt panqiwei nicole1130 meibaihui jackonyang wurentidai hupidong 1032998 napoler gumin2020 weihaoaho fword lichao88 hmrg-grmh ddl-donglin yang-zhikai zuiwufenghua yueyedeai suparek joytianya superxiang dlutoycchinese-electra's Issues
下游微调任务 CMRC 2018训练集和开发集
按照网站的提示，没有运行成功啊，json格式的数据转换成tfrecord的代码，已经写好了吗？还是需要自己加。是否我只要按照提示格式，放好数据直接运行吗？
我在finetune CMRC 2018时发现预处理得到的数据不及预期
我看了finetune/qa/qa_tasks.py中的代码，似乎与原版的electra一模一样。
官方的代码针对squad2.0的预处理是英文的。英文的预处理可以适配中文的qa数据吗？
我们在运行该仓库的代码finetune CMRC数据时。得到的example大致如下
{'task_name': 'cmrc2018', 'eid': 0, 'qas_id': 'TRAIN_186_QUERY_0', 'qid': None, 'question_text': '范廷颂是什么时候被任为主教的？', 'doc_tokens': ['范廷颂枢机（，），圣名保禄·若瑟（），是越南罗马天主教枢机。1963年被任为主教；1990年被擢升为天主教河内总教区宗座署理；1994年被擢升为总主教，同年年底被擢升为枢机；2009年2月离世。范廷颂于1919年6月15日在越南宁平省天主教发艳教区出生；童年时接受良好教育后，被一位越南神父带到河内继续其学业。范廷颂于1940年在河内大修道院完成神学学业。范廷颂于1949年6月6日在河内的主教座堂晋铎；及后被派到圣女小德兰孤儿院服务。1950年代，范廷颂在河内堂区创建移民接待中心以收容到河内避战的难民。1954年，法越战争结束，越南**共和国建都河内，当时很多天主教神职人员逃至越南的南方，但范廷颂仍然留在河内。翌年管理圣若望小修院；惟在1960年因捍卫修院的自由、自治及拒绝政府在修院设政治课的要求而被捕。1963年4月5日，教宗任命范廷颂为天主教北宁教区主教，同年8月15日就任；其牧铭为「我信天主的爱」。由于范廷颂被越南政府软禁差不多30年，因此他无法到所属堂区进行牧灵工作而专注研读等工作。范廷颂除了面对战争、贫困、被当局**天主教会等问题外，也秘密恢复修院、创建女修会团体等。1990年，教宗若望保禄二世在同年6月18日擢升范廷颂为天主教河内总教区宗座署理以填补该教区总主教的空缺。1994年3月23日，范廷颂被教宗若望保禄二世擢升为天主教河内总教区总主教并兼天主教谅山教区宗座署理；同年11月26日，若望保禄二世擢升范廷颂为枢机。范廷颂在1995年至2001年期间出任天主教越南主教团主席。2003年4月26日，教宗若望保禄二世任命天主教谅山教区兼天主教高平教区吴光杰主教为天主教河内总教区署理主教；及至2005年2月19日，范廷颂因获批辞去总主教职务而荣休；吴光杰同日真除天主教河内总教区总主教职务。范廷颂于2009年2月22日清晨在河内离世，享年89岁；其葬礼于同月26日上午在天主教河内总教区总主教座堂举行。'], 'orig_answer_text': '1963年', 'start_position': 0, 'end_position': 0, 'is_impossible': False}
该预处理似乎无法正确的进行token的切分以及start、end位置的查找。
预训练数据量
大神，我想问下，electra-small, electra-large预训练的训练数据大概是多少？
如何加载开源的Model，在其上再做预训练？
是将模型拷贝在./data-dir/models 里嘛？
多分类任务如何修改？
如果进行多分类情感分析的话，修改哪里呢？直接将CnSentiCorp任务里的["0","1"]修改为["1","2","3","4","5"]吗？
how to download a tf2.0 model of electra-hongkongese-large-discriminator ??
Thanks for your amazing work
I would like to use a bert4keras framework to funetine a model for ner task.
Therefore, i would like to download a electra HK large discriminator for tf2.0 to my local machine.
Thanks
Marcus
请问finetune时应如何设置token type id?
在Bert中若处理输入为两个句子的相关任务（例如语义相似性打分等），常使用token_type_embedding对两个句子分别加上不同的embedding；这一做法只需要在transformers的API中设置token_type_id（一个句子全为0，另一个全为1）即可实现。
然而，electra的预训练好像取消了NSP任务，相应的也就没有训练这个句子embedding（抱歉我不是很确定，只是看了一下论文好像没写这一点😂），所以我想请教一下使用token_type_id这一做法是否在electra中也可以通用呢？如果不行，对于两个句子的输入，推荐的处理方法是什么呢？谢谢！
预训练数据格式可以提供一下吗
IOError in tf.io.TFRecordWriter(output_file)
there is a bug in finetune/preprocessing.py
def _serialize_dataset(self, tasks, is_training, split):
utils.mkdir(tfrecords_path.rsplit("/", 1)[0])
this code used "/" as split flag to split out_file path ,if this code running in widows ,it might make the filename as directory ,then it will make IOError in method "def serialize_examples(self, examples, is_training, output_file, batch_size):" -->with tf.io.TFRecordWriter(output_file) as writer:
PLS use "utils.mkdir(tfrecords_path.rsplit(os.sep,1)[0])" insteadof "utils.mkdir(tfrecords_path.rsplit("/", 1)[0])"
最好还是配上对应的json
解压后发现没有json文件，而不少框架都是根据json文件来读取模型基本结构的，建议还是配上。
比如small版
{
"attention_probs_dropout_prob": 0.1,
"directionality": "bidi",
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 256,
"initializer_range": 0.02,
"intermediate_size": 1024,
"max_position_embeddings": 512,
"num_attention_heads": 4,
"num_hidden_layers": 12,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"type_vocab_size": 2,
"vocab_size": 21128,
"embedding_size": 128
}
还有，不知道为啥ckpt的命名不加上ckpt...
最后，最新版bert4keras（0.6.4）已经能加载electra了，只需要在build_transformer_model里边传入model='electra'，欢迎用bert4keras调用哈哈～
Sentiment classification
您好，很感谢您的分享，我想用ELECTRA-large做二分类的情感分类，该如何进行代码复现呢，目前我遇到了问题不知道怎么解决，望您指导，谢谢
在預訓練的時候是否也有使用全詞遮蔽？
之前bert-wwm可以改善原先bert預訓練mask單個字的問題，全詞遮蔽(wwm)可以使模型學到更多詞與詞的關係。
目前這一個版本的electra在預訓練的時候是否也有使用全詞遮蔽(wwm)?
finetune/classification/classification_metrics.py F1Scorer计算代码有误
finetune/classification/classification_metrics.py中：
def _get_results(self): 计算代码有误。
关于ELECTRA的预训练方法
ELECTRA由生成器+判决器构成，生成器负责把[MASK]替换成实际tokens，判决器负责区分替换结果是否和实际data中相同
但是，论文提及：
-
Typically k = [0.15n], i.e., 15% of the tokens are masked out
-
if the generator happens to generate the correct token, that token is considered “real” instead of “fake
也就是说，只有 (1 - generator_inference_acc) * 0.15的token会被预测成fake，训练到后期就是一个极度不均衡的二分类问题（对于判决器而言），为什么判决器不会受到影响呢？
运行TF版本的模型，提示python停止工作
没有GPU，每次运行时python就会提示停止工作
D:\paper\electra>python run_finetuning.py --data-dir \train --model-name electra
_small --hparams lcqmc.json
2020-06-28 19:51:11.486608: W tensorflow/stream_executor/platform/default/dso_lo
ader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64
_100.dll not found
2020-06-28 19:51:11.496134: I tensorflow/stream_executor/cuda/cudart_stub.cc:29]
Ignore above cudart dlerror if you do not have a GPU set up on your machine.
WARNING:tensorflow:From D:\paper\electra\model\optimization.py:70: The name tf.t
rain.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
请告知params_cmrc2018.json这个文件所在之处
您好：
我看了这个项目的README，不太清楚params_cmrc2018.json这个文件在哪里，烦请告知，非常感谢！
此致敬礼！
关于在领域数据集上继续预训练后微调效果很差的问题
你好，我在提供的base模型基础上用自己的数据继续预训练大约100w步，生成器loss 0.9左右，判别器loss 0.18左右，之后用预训练的模型在分类任务上微调，准确率只有10%+，而且用训练集测试准确率也只有30%。如果直接在提供的base模型基础上准确率能到90%，这是什么原因呢
ELECTRA-base的配置文件的参数设置
请问ELECTRA-base, Chinese: 12-layer, 768-hidden, 12-heads, 102M parameters的params_cmrc2018.json配置文件参数是什么？README里提供的是ELECTRA-small
请问pytorch版模型如何微调
请问pytorch版模型如何微调
small模型预训练参数
你好，我进行了small模型预训练，然后模型大小跟提供的不太一样，然后调用出现权重形状不兼容的错误，想问下预训练的参数这些，是多少？
请教，是否观察到 electra 较 bert/roberta 收敛更快？
比较 pretraining 不同 steps 的 checkpoint。同 step 对应的 checkpoint，electra 100% label 学习的优势，在 finetuning 效果上，论文里是显著快于 bert 的。
不知道复现是否有这个结论呢？我们在做一个类似的策略，收敛速度上并没有论文显著。
electra base的check point file在哪里？
该项目是已经预训练好的，预训练好的checkpoint file保存在哪里了？在运行fine tuning时报错未找到checkpoint file，烦请告知，感谢。
keyerror ：loss，我在自己的语料上finetune，training之后的evaluation阶段报错
self._total_loss +=results['loss'] keyerror:loss
huggingface的tokenizer问题
首先感谢作者所做的工作，我在使用过程中有两个疑问
- 请问ELECTRA可以使用huggingface的bert-tokenizer吗？词表完全相同是否能用相同的tokenizer?
- 请问Chinese-ELECTRA的预训练语料有多大呢?
ELECTRA-small, Chinese 权重丢了5个？
提供开源模型的老师，您好，我通过讯飞云下载的模型， 解析ELECTRA-small模型发现有五个权重找不到。
error electra/encoder/layer_9/intermediate/dense/kernel
error electra/encoder/layer_9/output/LayerNorm/beta
error electra/encoder/layer_9/output/LayerNorm/gamma
error electra/encoder/layer_9/output/dense/bias
error electra/encoder/layer_9/output/dense/kernel
请检查是否存在该问题，十分感谢。
换了个源下载是ok的，抱歉
download issue
你好，这里的ELECTRA-large, Chinese (new)不能正常下载，可以解决吗
如何进行测试？分类任务怎么得到每一个item的分类结果呢？
如何进行测试？分类任务怎么得到每一个item的分类结果呢？
请问预训练语料有没有格式的处理
用于预训练的语料有没有做类似于Bert或Albert的格式处理，我发现处理前后产生的tfrecord的样本量差距还是挺大的
对于 readme 的错误或疑问
readme中写的是：task-name：任务名称，本例中为cmrc2018。本目录中的代码已适配了以上六个中文任务，task-name分别为cmrc2018，drcd，xnli，chnsenticorp，lcqmc，bqcorpus。
但是代码里是： if task.name in ["cola", "mrpc", "mnli", "sst", "rte", "qnli", "qqp", "sts"]:
应该要将 readme 里的 xnli 改为 mnli 吧？
run_pretraining的hparams的参数能共享下吗？
在跑run_pretraining的时候碰到
logits = tf.matmul(hidden, model.get_embedding_table(),
transpose_b=True)
logits = tf.nn.bias_add(logits, output_bias)
这行报错，原因是hidden是的一个rank=3的tensor， model.get_embedding_table()是一个等于2的tensor，不知道有没有碰到这样的错误？
关于Loss计算的问题
您好，下游任务损失函数是加和（reduce_sum）计算，而非对Batch求均值（reduce_mean），感觉对部分非Adam优化器的结果会产生影响。想请教一下，这一细节是有意为之还是不会影响模型效果？此外非常感谢共享中文预训练模型。
CMRC 2018训练集和开发集输入数据格式对么？
代码中
input_data = json.load(f)["data"]
for entry in input_data:
for paragraph in entry["paragraphs"]
train.json 不是这个格式，直接跑不起来？
error in loading checkpoints for pretraining
error in loading checkpoints for pretraining, adam_m is missing?
2020-08-11 22:40:26.262591: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key discriminator_predictions/dense/bias/adam_m not found in checkpoint
ERROR:tensorflow:Error recorded from training_loop: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key discriminator_predictions/dense/bias/adam_m not found in checkpoint
[[node save/RestoreV2 (defined at run_pretraining.py:363) ]]
Original stack trace for 'save/RestoreV2':
File "run_pretraining.py", line 404, in <module>
main()
File "run_pretraining.py", line 400, in main
args.model_name, args.data_dir, **hparams))
File "run_pretraining.py", line 363, in train_or_eval
max_steps=config.num_train_steps)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1480, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 237, in finalize
self._saver.build()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
Traceback (most recent call last):
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key discriminator_predictions/dense/bias/adam_m not found in checkpoint
[[{{node save/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key discriminator_predictions/dense/bias/adam_m not found in checkpoint
[[node save/RestoreV2 (defined at run_pretraining.py:363) ]]
Original stack trace for 'save/RestoreV2':
File "run_pretraining.py", line 404, in <module>
main()
File "run_pretraining.py", line 400, in main
args.model_name, args.data_dir, **hparams))
File "run_pretraining.py", line 363, in train_or_eval
max_steps=config.num_train_steps)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1480, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 237, in finalize
self._saver.build()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1296, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1614, in object_graph_key_mapping
object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 678, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_pretraining.py", line 404, in <module>
main()
File "run_pretraining.py", line 400, in main
args.model_name, args.data_dir, **hparams))
File "run_pretraining.py", line 363, in train_or_eval
max_steps=config.num_train_steps)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2876, in train
rendezvous.raise_errors()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 131, in raise_errors
six.reraise(typ, value, traceback)
File "/home/test/anaconda3/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1480, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 647, in create_session
init_fn=self._scaffold.init_fn)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/session_manager.py", line 290, in prepare_session
config=config)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/session_manager.py", line 220, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1302, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key discriminator_predictions/dense/bias/adam_m not found in checkpoint
[[node save/RestoreV2 (defined at run_pretraining.py:363) ]]
Original stack trace for 'save/RestoreV2':
File "run_pretraining.py", line 404, in <module>
main()
File "run_pretraining.py", line 400, in main
args.model_name, args.data_dir, **hparams))
File "run_pretraining.py", line 363, in train_or_eval
max_steps=config.num_train_steps)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
saving_listeners)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1480, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1007, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1200, in __init__
_WrappedSession.__init__(self, self._create_session())
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1205, in _create_session
return self._sess_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 871, in create_session
self.tf_sess = self._session_creator.create_session()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 237, in finalize
self._saver.build()
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/test/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
自己的文本多分类任务，训练时展示没有acc，只有loss?
只有loss 没有acc 作为参考
关于预训练，步骤可以写的详细一点吗？
比如 ：
1）数据格式
2）运行代码步骤
多谢！
请问后续会有Large的模型么?
目前只有small和base版本的，请问后续会放Large版本的么？如果有，大概什么时候呢？
模型加载兼容问题
我感觉electra的modeling即是bert的modeling,预训练的优化在于优化目标；我想问下electra的run_finetuning.py能不能直接加载bert,bert-wwm，robertra等等的模型；目前加载似乎不太合适，主要修改那个代码块？
finetuning训练没有运行在GPU上
CUDA_VISIBLE_DEVICES=1 python run_finetuning.py xxxx
程序没有运行在GPU上，信息如下：
这个是什么原因？
想加载已有模型进一步预训练时缺少adam应该怎么办呢？
谷歌原repo提到对已有模型经一部预训练的方法是将路径改到已有模型上继续运行run_pretraining.py，即
Setting the model-name to point to a downloaded model (e.g., --model-name electra_small if you downloaded weights to $DATA_DIR/electra_small).
但使用中文electra时似乎因为去掉了adam_m而无法进一步预训练
报错信息为：Key discriminator_predictions/dense/bias/adam_m not found in checkpoint
如何用于生成类的任务
关于tensorflow版的发布的预训练模型
请问这里开源的tensorflow版本的预训练模型应该是D吧？
转换为transformers的格式报错
英文原版的可以转换
转换脚本：https://github.com/huggingface/transformers/blob/electra/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py
报错信息：
Traceback (most recent call last): packages/transformers/model_electra/modeling_electra.py", line 102, in load_tf_weights_in_electra assert pointer.shape == array.shape, original_name AssertionError: ('electra/embeddings/LayerNorm/beta', torch.Size([128]), (768,))
询问项目入口文件与项目启动命令
请问该项目的入口文件是哪一个？项目启动命令是什么？烦请告知，感谢！
关于预训练数据的格式，可以提供一下吗
keyError:3200000
Traceback (most recent call last):
File "run_finetuning.py", line 375, in
main()
File "run_finetuning.py", line 371, in main
args.model_name, args.data_dir, **hparams))
File "run_finetuning.py", line 304, in run_finetuning
scorer.write_predictions()
File "/Chinese-ELECTRA-master/finetune/qa/qa_metrics.py", line 113, in write_predictions
result = unique_id_to_result[feature[self._name + "_eid"]]
KeyError: 3200000
关于并行
您好，我在8个2080Ti上进行预训练，只有一个GPU显存几乎占满，另外7个都是只是用100M左右，加大batch_size就出现OOM问题了，请问代码中，有设置GPU使用的超参数吗，看了一遍没有找到。
代码的支持的TensorFlow版本是什么？
请问ELECTRA预训练会有loss降到一定程度反弹的现象嘛？MLM与disc_loss最低可以降到多少呢
可以提供一下预训练的配置和耗时信息吗？
我在chinese_electra_base的基础上，在自己的语料上继续预训练。发现训练速度特别慢。一个多小时200步。
我用的设备是tpu-v3。希望可以得到你们预训练的经验分享。谢谢
自己的文本分类数据用1e-5的lr微调，才可以收敛
ELECTRA在自己的文本分类数据上微调，用1e-5的lr才能收敛,用5e-5得到的准确率只有0.12，很奇怪。但是在RoBERTa调整lr之后效果不会差这么大
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.