jayparks / transformer Goto Github PK
View Code? Open in Web Editor NEWA Pytorch Implementation of "Attention is All You Need" and "Weighted Transformer Network for Machine Translation"
A Pytorch Implementation of "Attention is All You Need" and "Weighted Transformer Network for Machine Translation"
In the original paper(weighted transformer), the author mentioned that "all bounds are respected during each training step by projection."
I have no idea what "by project" means and don't know how to keep the constrains of sum(k)=1 and sum(α)=1.
It seems there is no particular processing in this repository except for initialization. Could you please explain?
This error occurs when I run train.py
Traceback (most recent call last):
File "train.py", line 208, in
main(opt)
File "train.py", line 72, in main
train_loss, train_sents = train(model, criterion, optimizer, train_iter, model_state)
File "train.py", line 113, in train
dec_inputs, dec_inputs_len)
File "C:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "D:\transformer-master\transformer\models.py", line 152, in forward
enc_outputs, enc_self_attns = self.encoder(enc_inputs, enc_inputs_len, return_attn)
File "C:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "D:\transformer-master\transformer\models.py", line 59, in forward
enc_outputs = self.src_emb(enc_inputs)
File "C:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "C:\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py", line 110, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "C:\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1110, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument #3 'index'
......?
TypeError Traceback (most recent call last)
in
2 d_k = 16
3 n_heads = 6
----> 4 w_q = Linear([d_model, d_k * n_heads])
5 w_q
TypeError: init() missing 1 required positional argument: 'out_features'
thx for your sharing, but I raise this error, can you give me some advice?
Hi. Please i would like to add some features to this code that i have read on beam search. Like coverage penalty and length normalization. But i don't know where to start. Can you help please?
''' train.py line 104
enc_inputs, enc_inputs_len = batch.src
dec_, dec_inputs_len = batch.trg
dec_inputs = dec_[:, :-1]
dec_targets = dec_[:, 1:]
dec_inputs_len = dec_inputs_len - 1
'''
In the original paper of Transformer, the input of Decoder is the earlier outputs but not labels.
When I run the colder, there is an error in model.py (line 53):
self.layers = nn.ModuleList( [self.layer_type(d_k, d_v, d_model, d_ff, n_heads, dropout) for _ in range(n_layers)])
only integer tensors of a single element can be converted to an index
How can I fix it
Please i would like to know how to use this repository because i am getting errors that i don't understand will running it.
The error "ValueError: only one element tensors can be converted to Python scalars" occurred in L79: input_pos = tensor([list(range(1, len+1)) + [0]*(max_len-len) for len in input_len]) in modules.py.
I want to know how to fix it, very grateful to get your reply!
The test set is useless and there are lots of bugs....
python3 train.py -model_path models -data_path models/preprocess-train.t7
Namespace(batch_size=128, d_ff=2048, d_k=64, d_model=512, d_v=64, data_path='models/preprocess-train.t7', display_freq=100, dropout=0.1, log=None, lr=0.0002, max_epochs=10, max_grad_norm=None, max_src_seq_len=50, max_tgt_seq_len=50, model_path='models', n_heads=8, n_layers=6, n_warmup_steps=4000, share_embs_weight=False, share_proj_weight=False, weighted_model=False)
Loading training and development data..
Creating new model parameters..
Traceback (most recent call last):
File "train.py", line 200, in
main(opt)
File "train.py", line 47, in main
model, model_state = create_model(opt)
File "train.py", line 27, in create_model
model = Transformer(opt) # Initialize a model state.
File "/media/vivien/A/NEW-SMT/transformer-new-master/transformer/models.py", line 110, in init
opt.max_src_seq_len, opt.src_vocab_size, opt.dropout, opt.weighted_model)
File "/media/vivien/A/NEW-SMT/transformer-new-master/transformer/models.py", line 54, in init
[self.layer_type(d_k, d_v, d_model, d_ff, n_heads, dropout) for _ in range(n_layers)])
File "/media/vivien/A/NEW-SMT/transformer-new-master/transformer/models.py", line 54, in
[self.layer_type(d_k, d_v, d_model, d_ff, n_heads, dropout) for _ in range(n_layers)])
File "/media/vivien/A/NEW-SMT/transformer-new-master/transformer/layers.py", line 11, in init
self.enc_self_attn = MultiHeadAttention(d_k, d_v, d_model, n_heads, dropout)
File "/media/vivien/A/NEW-SMT/transformer-new-master/transformer/sublayers.py", line 53, in init
self.multihead_attn = _MultiHeadAttention(d_k, d_v, d_model, n_heads, dropout)
File "/media/vivien/A/NEW-SMT/transformer-new-master/transformer/sublayers.py", line 19, in init
self.w_q = Linear([d_model, d_k * n_heads])
TypeError: init() missing 1 required positional argument: 'out_features'
how to use it ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.