harvardnlp / annotated-transformer Goto Github PK
View Code? Open in Web Editor NEWAn annotated implementation of the Transformer paper.
Home Page: http://nlp.seas.harvard.edu/annotated-transformer
License: MIT License
An annotated implementation of the Transformer paper.
Home Page: http://nlp.seas.harvard.edu/annotated-transformer
License: MIT License
occurs in the block immediately after the cell with the comment
criterion = LabelSmoothing(size=len(TGT.vocab), padding_idx=pad_idx, smoothing=0.1)
criterion.cuda()
for epoch in range(15):
train_epoch((rebatch(pad_idx, b) for b in train_iter), model, criterion, model_opt)
valid_epoch((rebatch(pad_idx, b) for b in valid_iter), model, criterion)
TypeError Traceback (most recent call last)
in ()
2 criterion.cuda()
3 for epoch in range(15):
----> 4 train_epoch((rebatch(pad_idx, b) for b in train_iter), model, criterion, model_opt)
5 valid_epoch((rebatch(pad_idx, b) for b in valid_iter), model, criterion)
2 frames
/usr/local/lib/python3.6/dist-packages/torchtext/data/iterator.py in iter(self)
149 minibatch.sort(key=self.sort_key, reverse=True)
150 yield Batch(minibatch, self.dataset, self.device,
--> 151 self.train)
152 if not self.repeat:
153 return
TypeError: init() takes from 1 to 4 positional arguments but 5 were given
Hello!
Thank you very much for your contribution.
I wonder how to adapt the code in order to train a model on IWSLT data on my PC without GPUs.
It seems like MultiGPULossCompute
should be replaced in run_epoch
, but SimpleLossCompute
doesn't seem like an appropriate candidate.
I would appreciate any hint.
All the images in the google colab version of the notebook won't render.
Heres' the link to the google colab version:
https://colab.research.google.com/drive/1xQXSv6mtAOLXxEMi8RvaW8TW-7bvYBDF
because of Noam Shazeer, one of the authors of the paper?
what is the connection?
Why use
subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')
return torch.from_numpy(subsequent_mask) == 0
I think
return torch.from_numpy(np.tril(np.ones(attn_shape), k=0).astype('uint8'))
maybe is a more clear writing style for a new learner?
when I run this code, there occurred some problems
(1) in LabelSmoothing class
this line => true_dist.index_fill_(0, mask.squeeze(), 0.0)
RuntimeError: invalid argument 3: Index is supposed to be a vector at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/TH/generic/THTensorMath.cpp:569
(2) in SimpleLossCompute class
this lien = > loss = self.criterion(x.contiguous().view(-1, x.size(-1)), y.contiguous().view(-1)) / norm
RuntimeError: Expected object of type torch.FloatTensor but found type torch.LongTensor for argument #2 'other'
my torch version is 0.4.1
does anyone have these problem? thanks~
def rate(self, step = None):
"Implement `lrate` above"
if step is None:
step = self._step
return self.factor * \
(self.model_size ** (-0.5) *
min(step ** (-0.5), step * self.warmup ** (-1.5)))
What's the role of self.factor?
I was following your annotated Ipython code, and some weird error came up during data loading section.
.data/iwslt/de-en/IWSLT16.TED.tst2012.de-en.en.xml
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-74-c069794b494b> in <module>()
23 train, val, test = datasets.IWSLT.splits(
24 exts=('.de', '.en'), fields=(SRC, TGT),
---> 25 filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and
26 len(vars(x)['trg']) <= MAX_LEN)
27 MIN_FREQ = 2
/usr/local/lib/python2.7/dist-packages/torchtext-0.2.1-py2.7.egg/torchtext/datasets/translation.pyc in splits(cls, exts, fields, root, train, validation, test, **kwargs)
136
137 if not os.path.exists(os.path.join(path, train) + exts[0]):
--> 138 cls.clean(path)
139
140 train_data = None if train is None else cls(
/usr/local/lib/python2.7/dist-packages/torchtext-0.2.1-py2.7.egg/torchtext/datasets/translation.pyc in clean(path)
156 for doc in root.findall('doc'):
157 for e in doc.findall('seg'):
--> 158 fd_txt.write(e.text.strip() + '\n')
159
160 xml_tags = ['<url', '<keywords', '<talkid', '<description',
TypeError: write() argument 1 must be unicode, not str
It seems like the code cannot locate the path of the datasets, although the dataset is actually in the path.
Was there any update on dependencies after this code released?
In the text (between cells 7 and 8), you say the output of each sub-layer is:
layer_norm(x + dropout(sublayer(x)))
but in the code it looks like it's implemented as
x + dropout(sublayer(layer_norm(x))
Am I understanding correctly? (I'm guessing this doesn't matter in practice, but just checking)
Thanks
Thank you for a great piece.
I have a question about the method foward in LabelSmoothing. I debug it by Pycharm and the excuting order is from
loss = self.criterion(x.contiguous().view(-1, x.size(-1)), #[30, 9, 512] --> [270, 512]
y.contiguous().view(-1)) / norm #[30, 9] --> [270] (SimpleLossCompute's method call)
to
method forward in LabelSmoothing
def forward(self, x, target): #x is the output of model, target is label
assert x.size(1) == self.size
true_dist = x.data.clone()
....
why the dim of x changes from [270, 512] to [270, 11]
I installed pytorch 0.3 as requested and also in python 1.1 and it does not work.
I mean even the toy example does not work.
This significantly diminish value of this implementation, so I suggest to fix the code so it works with never version of pytorch.
Or fix it so it work in any version of pytorch :)
I think this line in the MultiHeadedAttention: l(x).view(nbatches, -1, self.h, self.d_k)
should be l(x).view(nbatches, self.h, -1, self.d_k)
, such that the unsqueezed mask could be applied to all heads.
when split dataset with this following line ,the error happens:
train, val, test = datasets.IWSLT.splits(
exts=('.de', '.en'), fields=(SRC, TGT),
filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and
len(vars(x)['trg']) <= MAX_LEN)
I think that's mostly due to the versions' incompatible
my versions(actually there is not a requirements file in the repo,I'll be gratefull if someone add it ):
torch==0.3.0.post4
torchtext==0.2.3
spacy==2.0.10
torchvision==0.2.1
thank u!
to run on pytorch 1.0 both the position and the div term need to be initialized as float instead of int in the PositionalEncoding Class
ie.
position = torch.arange(0.0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0.0, d_model, 2) *
-(math.log(10000.0) / d_model))
in lieu of
torch.arange(0, ...
thank you for a great breakdown of Vaswani's paper
Traceback (most recent call last):
File "C:/Users/01/Desktop/机器学习作业/AllenNLP/test.py", line 632, in
train_epoch(data_gen(V, 30, 20), model, criterion, model_opt)
File "C:/Users/01/Desktop/机器学习作业/AllenNLP/test.py", line 589, in train_epoch
out = model.forward(src, trg[:, :-1], src_mask, trg_mask[:, :-1, :-1])
File "C:/Users/01/Desktop/机器学习作业/AllenNLP/test.py", line 42, in forward
return self.decode(self.encode(src, src_mask), src_mask, tgt, tgt_mask)
File "C:/Users/01/Desktop/机器学习作业/AllenNLP/test.py", line 36, in encode
return self.encoder(self.tgt_embed(src), src_mask)
File "C:\ProgramData\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "C:\ProgramData\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\container.py", line 91, in forward
input = module(input)
File "C:\ProgramData\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "C:/Users/01/Desktop/机器学习作业/AllenNLP/test.py", line 319, in forward
return self.lut(x) * math.sqrt(self.d_model)
File "C:\ProgramData\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "C:\ProgramData\Anaconda3\envs\python36\lib\site-packages\torch\nn\modules\sparse.py", line 108, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "C:\ProgramData\Anaconda3\envs\python36\lib\site-packages\torch\nn\functional.py", line 1076, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CPUIntTensor instead (while checking arguments for embedding)
When I using the Colab, I got the correct result.
But on my own Pycharm, I got this problem.
Thx, this model is new to me.
Can you please provide the dataset that you had pre-processed for the translation task?
Why not simply with the original expression of the paper?
I know the expression in the code as below is equivalent to the original one:
div_term = torch.exp(torch.arange(0, d_model, 2) *-(math.log(10000.0) / d_model))
Just like what srush said in #1
BUT, is that won't be a problem? since you also normalized the input embedding by coding like that.
I am consistently getting two errors which I have failed to resolve even after spending hours on it!
Require some serious help here!!
.
.
.
<ipython-input-139-2ae4ba63671c> in encode(self, src, src_mask)
17
18 def encode(self, src, src_mask):
---> 19 return self.encoder(self.src_embed(src), src_mask)
20
21 def decode(self, memory, src_mask, tgt, tgt_mask):
.
.
.
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
.
.
.
74 def encode(self, src, src_mask):
---> 75 return self.encoder(self.src_embed(src), src_mask)
Both has self.encoder() as fault. I can't really figure out what is happening. It will super great if someone can provide insights on this. Link to that jupyter notebook:
https://github.com/AmoghM/DeepLearning/blob/master/TransformerNetwork/HarvardTransformer.ipynb
TypeError Traceback (most recent call last)
in ()
9 model.train()
10 run_epoch(data_gen(V, 30, 20), model,
---> 11 SimpleLossCompute(model.generator, criterion, model_opt))
12 model.eval()
13 print(run_epoch(data_gen(V, 30, 5), model,
in run_epoch(data_iter, model, loss_compute)
7 for i, batch in enumerate(data_iter):
8 out = model.forward(batch.src, batch.trg,
----> 9 batch.src_mask, batch.trg_mask)
10 loss = loss_compute(out, batch.trg_y, batch.ntokens)
11 total_loss += loss
in forward(self, src, tgt, src_mask, tgt_mask)
14 def forward(self, src, tgt, src_mask, tgt_mask):
15 "Take in and process masked src and target sequences."
---> 16 return self.decode(self.encode(src, src_mask), src_mask,
17 tgt, tgt_mask)
18
in encode(self, src, src_mask)
18
19 def encode(self, src, src_mask):
---> 20 return self.encoder(self.src_embed(src), src_mask)
21
22 def decode(self, memory, src_mask, tgt, tgt_mask):
X:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in call(self, *input, **kwargs)
323 for hook in self._forward_pre_hooks.values():
324 hook(self, input)
--> 325 result = self.forward(*input, **kwargs)
326 for hook in self._forward_hooks.values():
327 hook_result = hook(self, input, result)
X:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\container.py in forward(self, input)
65 def forward(self, input):
66 for module in self._modules.values():
---> 67 input = module(input)
68 return input
69
X:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in call(self, *input, **kwargs)
323 for hook in self._forward_pre_hooks.values():
324 hook(self, input)
--> 325 result = self.forward(*input, **kwargs)
326 for hook in self._forward_hooks.values():
327 hook_result = hook(self, input, result)
in forward(self, x)
6
7 def forward(self, x):
----> 8 return self.lut(x) * math.sqrt(self.d_model)
X:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\module.py in call(self, *input, **kwargs)
323 for hook in self._forward_pre_hooks.values():
324 hook(self, input)
--> 325 result = self.forward(*input, **kwargs)
326 for hook in self._forward_hooks.values():
327 hook_result = hook(self, input, result)
X:\ProgramData\Anaconda3\Lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
101 input, self.weight,
102 padding_idx, self.max_norm, self.norm_type,
--> 103 self.scale_grad_by_freq, self.sparse
104 )
105
X:\ProgramData\Anaconda3\Lib\site-packages\torch\nn_functions\thnn\sparse.py in forward(cls, ctx, indices, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
57 output = torch.index_select(weight, 0, indices)
58 else:
---> 59 output = torch.index_select(weight, 0, indices.view(-1))
60 output = output.view(indices.size(0), indices.size(1), weight.size(1))
i was trying to run this code in Jupyter notebook,but when i run this cell, it came out an error: 'The kernel appears to have died. It will restart automatically.' I cant figure out why this error will come out,could anybody offer me some help? Thank you so much.
# Train the simple copy task.
V = 11
criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)
model = make_model(V, V, N=2)
model_opt = NoamOpt(model.src_embed[0].d_model, 1, 400,
torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))
for epoch in range(3):
model.train()
run_epoch(data_gen(V, 30, 20), model,
SimpleLossCompute(model.generator, criterion, model_opt))
model.eval()
print(run_epoch(data_gen(V, 30, 5), model,
SimpleLossCompute(model.generator, criterion, None)))
class SublayerConnection(nn.Module):
def __init__(self, size, dropout):
super(SublayerConnection, self).__init__()
self.norm = LayerNorm(size)
self.dropout = nn.Dropout(dropout)
def forward(self, x, sublayer):
"Apply residual connection to any sublayer with the same size."
#return x + self.dropout(sublayer(self.norm(x)))
return self.norm(x + self.dropout(sublayer(x)))
In the paper, it seems to be "return self.norm(x + self.dropout(sublayer(x)))". But it gives a bad result for the example "Train the simple copy task."
When I run the case of class LabelSmoothing, I met a problem showns as follows:
RuntimeError: invalid argument 3: Index is supposed to be a vector at d:\build\pytorch\pytorch-0.4.1\aten\src\th\generic\thtensormath.cpp:569.
I find this line "true_dist.index_fill_(0, mask.squeeze(), 0)" cause the error because mask is blank.
Besides, the second case still has problem. The second case is :
crit = LabelSmoothing(5, 0, 0.4)
predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],
[0, 0.2, 0.7, 0.1, 0],
[0, 0.2, 0.7, 0.1, 0]])
v = crit(Variable(predict.log()),
Variable(torch.LongTensor([2, 1, 0])))
plt.imshow(crit.true_dist)
None
The result of v is "inf".
Does anyone have the same problem?
I am wandering whether the label smoothing works efficiently. How about deleting the label smoothing process in the project.
Hope your reply. Thanks a lot!
Hi, thanks much for sharing. I have a quick question in the loss calculation for mutiplegpu. I am not very familiar with pytorch and having hard time understanding this line : o1.backward(gradient=o2)
, why backprop is done on out
(not loss) and why we need to pass o2
?, thanks for your answer.
Hello,
Thanks for the awesome work. I am trying to adapt this to my own dataset but having trouble in understanding the batch size used and the iterator as I am new to torchtext. The batch size is set as 12000 in the example but when I try to print the tensor sizes within the batch, the shapes don't really reflect that. Am I misunderstanding something?
BATCH_SIZE = 12000
train_iter = MyIterator(train, batch_size=BATCH_SIZE, device=0,
repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),
batch_size_fn=batch_size_fn, train=True)
i = 0
for b in train_iter:
i+=1
print("steps: " + str(i))
for b in train_iter:
print(b.src.size())
print(b.trg.size())
return
"""
output
steps: 402
torch.Size([31, 203])
torch.Size([59, 203])
"""
Sorry for my ignorance. But how should I interpret this? Does the step equal to total data size / batch size?
Thank you very much for your help
There's a problem about the vocabulary of https://s3.amazonaws.com/opennmt-models/iwslt.pt : After loading the model by model = torch.load("iwslt.pt")
, it can be found that size of the English vocabulary is 36321
(iwslt.pt : size 36321; built from datasets.IWSLT: 36327)
(0): Embeddings(
(lut): Embedding(36321, 512)
)
However, after building the TGT.vocab by
MAX_LEN = 100
train, val, test = datasets.IWSLT.splits(
exts=('.de', '.en'), fields=(SRC, TGT),
filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and
len(vars(x)['trg']) <= MAX_LEN)
MIN_FREQ = 2
SRC.build_vocab(train.src, min_freq=MIN_FREQ)
TGT.build_vocab(train.trg, min_freq=MIN_FREQ)
, it's found that the size of the English vocabulary is
print("vocab_size = ", len(TGT.vocab) ) # 36327
print("vocab_size = ", len(TGT.vocab.itos) ) # 36327
print("vocab_size = ", len(TGT.vocab.stoi) ) # 36327
The codes for building vocab are almost identical. Perhaps datasets.IWSLT has changed slightly so that the vocab differs slightly.
Though translation results of the model on the 'valid_iter' seem quite correct, the model loaded from 'iwslt.pt' still cannot fully work since the vocabulary currently built from datasets.IWSLT does not match the vocabulary size.
I am building a model based on the iwslt.pt so I need the English vocabulary. How or where to obtain the correct English vocabulary of 'https://s3.amazonaws.com/opennmt-models/iwslt.pt' (size 36321) ?
Since we have
model_par = torch.nn.DataParallel(model, devices=devices)
do we need MultiGPULossCompute?
I thought pytorch comes with targets scattering and other stuff inside MultiGPULossCompute prebuilt in DataParallel.
I faced with some empty error while using MultiGPULossCompute, so I went with SimpleLossCompute and everything was fine.
Hello. What license is the code/post under?
Hello again!
When I tried to load the pre-trained model
model = torch.load("iwslt.pt")
it fails with the error message
RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:107
Is it possible (in addition to the entire model) to save just model weights so that I can load the model on any platform (it can be done with something like torch.save(the_model.state_dict(), PATH)
)?
Running the code locally gets me an error about shapes not being broadcastable, although when I run in the notebook, I don't get the error, and printing the shapes reveals that they aren't broadcastable. What's going on?
How to compute the BLEU score of the models implemented in this repo for the provided examples?
I have a question regarding the decoding loop (cell 45)
for i, batch in enumerate(valid_iter):
I want to run for several validation instances not for only one. Even when I deleted break
from the end of the loop It still fails after the first instance with error message
DeprecationWarning: generator 'Iterator.__iter__' raised StopIteration
When I replace valid_iter
with train_iter
. It runs for the first 42 instances then fails with the same error message.
I appreciate your help.
thanks for this tuorial ,it's really helpful to me,but i can't find where is the different cased by teacher forcing between trainning process and inference process ,thanks
the forward() in class LabelSmoothing throws a bug when run the block [29], can you check the code of condition judgement "if mask.dim() > 0:" should be "if mask.nelement() > 0:" .because if the mask is tensor([]), then the dim() is equal 1, and number of element is equal 0.
the following is my modified code, then there is no bug like before.
def forward(self, x, target):
assert x.size(1) == self.size
true_dist = x.data.clone()
true_dist.fill_(self.smoothing / (self.size - 2))
true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
true_dist[:, self.padding_idx] = 0
mask = torch.nonzero(target.data == self.padding_idx)
if mask.nelement() > 0:
true_dist.index_fill_(0, mask.squeeze(), 0.0)
self.true_dist = true_dist
return self.criterion(x, Variable(true_dist, requires_grad=False))
Training on CPU with:
torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
python 3.5
I have changed the current code as it was throwing warning and error with torch 1.+ as follows:
torch.arange(0, max_len) to torch.arange(0.0, max_len)
torch.exp(torch.arange(0, d_model, 2) to torch.exp(torch.arange(0.0, d_model, 2)
nn.KLDivLoss(size_average=False) to nn.KLDivLoss(reduction='sum')
nn.init.xavier_uniform(p) to nn.init.xavier_uniform_(p)
Variable(torch.LongTensor([1]))).data[0] to Variable(torch.LongTensor([1])))
SimpleLossCompute
loss.data[0] to loss
Also I deleted the In[37] and In[43] as I am training with CPU only.
Now, when I train it, it's showing the following error:
Epoch Step: 1 Loss: 3.141923 Tokens per Sec: 540.000000
Floating point exception (core dumped)
In
class EncoderLayer(nn.Module):
def forward(self, x, mask):
x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))
x is repeated 3 times for getting k,q,v
In
class MultiHeadedAttention(nn.Module):
def __init__(self, h, d_model, dropout=0.1):
self.d_k = d_model // h
...
def forward(self, query, key, value, mask=None):
# 1) Do all the linear projections in batch from d_model => h x d_k
query, key, value = \
[l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)
for l, x in zip(self.linears, (query, key, value))]
there is l(x).view(nbatches, -1, self.h, self.d_k)
which reshape data into self.h
heads.
This indicates the input should already have a dimension of number of heads.
Question/Confusion: I did not find where original input x is copied self.h times.
In class Batch, those mask about padding in sentence confuses me a lot. By conducting on how src_mask and tgt_mask working, I find that those masks only mask columns when doing self-attention after Q x K. However, I think both columns and rows of the calculated square matrix should be masked. Can anyone help me to figure it out? Thanks.
Sorry for raising another issue again. I am trying to adapt this code to my own dataset. Everything works fine except for loss computation with multiple gpus, with an assertion error in parallel_apply. Any idea how to fix this? I am using pytorch 0.4.1. Thank you so much for your help
train_loss = run_epoch((rebatch(pad_idx, b) for b in train_iter), model_par, MultiGPULossCompute(model.generator, criterion, devices=devices, opt=model_opt)) File "main.py", line 75, in run_epoch loss = loss_compute(out, batch.trg_y, batch.ntokens) File "/home/richie/ErrorCorrection/optim.py", line 59, in __call__ gen = nn.parallel.parallel_apply(generator, out_column) File "/home/richie/anaconda3/envs/python36/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 30, in parallel_apply assert len(modules) == len(inputs) AssertionError
The MathJax cdn link uses HTTP, not HTTPS, and thus Chrome and other browsers will refuse to load it when the main page is loaded using HTTPS
When I run the case of class LabelSmoothing, I met a problem showns as follows:
RuntimeError: invalid argument 3: Index is supposed to be a vector at d:\build\pytorch\pytorch-0.4.1\aten\src\th\generic\thtensormath.cpp:569.
I find this line "true_dist.index_fill_(0, mask.squeeze(), 0)" cause the error because mask is blank.
Besides, the second case still has problem. The second case is :
crit = LabelSmoothing(5, 0, 0.4)
predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],
[0, 0.2, 0.7, 0.1, 0],
[0, 0.2, 0.7, 0.1, 0]])
v = crit(Variable(predict.log()),
Variable(torch.LongTensor([2, 1, 0])))
plt.imshow(crit.true_dist)
None
The result of v is "inf".
Does anyone have the same problem?
I am wandering whether the label smoothing works efficiently. How about deleting the label smoothing process in the project.
Hope your reply. Thanks a lot!
In the __call__
function of class SimpleLossCompute
, loss.backward()
is called even if opt
is None, i.e. during validation. Wouldn't this cause gradients to be computed on the validation set and thus you are training on the validation data?
It would be a good thing to have / would simplify reading.
The code duplicate encoder/decoder layers using the following clones
function:
def clones(module, N):
"Produce N identical layers."
return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])
My understanding is that this would cause all layers to share the same random initial values for the learnable parameters, which is likely non-optimal. Is this just to make the code simpler? I feel like there are easy alternatives, e.g.:
def clones(module_fn, N):
"Produce N identical layers."
return nn.ModuleList([module_fn() for _ in range(N)])
On a different notes, kudos for writing such a clear and useful document!
Any plan to update?
Thanks for providing this fine-grained tutorial on reproducing Transformer in Pytorch!
I'm wondering if you have any plan on adding the support for beam search? Thanks!
Best
@srush Thank you so much for this post. However it will be great if you can help me with following clarification regarding Positional Encoding.
The whole intent of using positional encoding is to bring a sense of Positions (absolute or relative) and Time. Using Sine(for even positions) and Cos (for odd positions) wave, how do we embed this?
Also,
I'm looking at the section "Attention Visualization" in the jupyter notebook: "The Annotated Transformer.ipynb".
Is there any chance that there is a mistake in the visualization code?
When presenting the "Decoder Src Layer", the code that extracts the attention scores from the "self_attn" layers is:
fig, axs = plt.subplots(1,4, figsize=(20, 10)) for h in range(4): draw(model.decoder.layers[layer].self_attn.attn[0, h].data[:len(tgt_sent), :len(sent)], sent, tgt_sent if h ==0 else [], ax=axs[h])
Shouldn't it be replaced by the "src_attn" layer instead of the "self_attn" layer?
Thanks,
Ori
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.