Comments (3)
刚看了下,浅答一下,有什么问题的话,望大佬轻喷
paper :https://arxiv.org/pdf/2011.05864.pdf
要看懂这个论文,主要看下图1,公式4。说的很清楚了,将 BERT sentence embeddings 映射到 标准高斯分布空间。
看下这几个地方的代码:objective_tower、top_prior
无监督的话主要看下model_fn_builder
from bert-flow.
大佬,有监督 和 无监督 的 训练目标 各是什么啊?
from bert-flow.
目标是一致的,无监督和有监督在这里的区别就是是否把监督学习的 loss 算进去,model_fn_builder。
训练的是向量空间映射函数,假设这个函数是可逆的,增加了一个flow来学习这个函数。(上个评论已经说很清楚了)。
你具体要问的东西可能是无监督具体在学什么,应该在 objective_tower 函数。
with arg_scope(ops, init=init):
encoder = glow_ops.encoder_decoder
self.z, encoder_objective, self.eps, _, _ = encoder(
"flow", x, self.hparams, eps=None, reverse=False)
objective += encoder_objective
self.z_top_shape = get_shape_list(self.z)
prior_dist = self.top_prior()
prior_objective = tf.reduce_sum(
prior_dist.log_prob(self.z), axis=[1, 2, 3])
#self.z_sample = prior_dist.sample()
objective += prior_objective
# bits per pixel
_, h, w, c = get_shape_list(x)
objective = -objective / (np.log(2) * h * w * c)
主要看下这几行
from bert-flow.
Related Issues (19)
- 请问, 校正过程所用语料覆盖面是否要足够大 HOT 7
- Is there a pytorch version coeds in the future? HOT 1
- The paper link at the beginning of README is incorrect HOT 1
- 无监督训练为什么会有label呢? HOT 5
- some questions about this paper
- predict 的问题 HOT 3
- loss为负 HOT 15
- about flow acc HOT 5
- Will it be better to replace BERT with ELECTRA?
- Fine-tuning bert-flow embedding with supervision
- avg-last-2 取的是[-0]和[-1]层的embedding HOT 2
- How can I get a single sentence embedding? HOT 4
- Which code show that flow network is optimized while the BERT parameters remain unchanged?
- May I get access to BERT-base-NLI-flow ?
- do you have pytorch implement? HOT 1
- Correction for a mistake in 'last2avg'
- 训练损失为负数,
- 各位大佬,有监督 和 无监督 的 训练目标 各是什么啊?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bert-flow.