Hi

Does the code run on CPU?

I run this model on 1080ti
command: python train.py --dataset rocstories --desc rocstories --submit --analysis
Then I got ResourceExhaustedError
Do I need to make some changes to the code before running?
Thanks.

    if rf == 1:  # faster 1x1 conv
       c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf])) + b, shape_list(x)[:-1] + [nf])
    else:  # was used to train LM
        c = tf.nn.conv1d(x, w, stride=1, padding=pad) + b  

when training LM, rf = ? and why?

hi
how can I solve this error??

OSError: [E050] Can't find model 'en'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.

Anybody can help me??

can we add the classification layers in the model and train it for some steps for a perfect classification?
** also please share a data set for the classification task i want.

From what I can tell, for the RoCstories code, M is initialized to 1 here:
https://github.com/openai/finetune-transformer-lm/blob/master/train.py#L243

And the only place where it's used in the model (after a reshape) is here:
https://github.com/openai/finetune-transformer-lm/blob/master/train.py#L179

What is M supposed to represent/encode?

n attention class, you have the following code for masking. I understand the logic for pre training, but in fine tuning if we dont include language model loss we should have a check here for not applying the mask. Do we have to always apply the masking because the model was trained that way, is there an intuitive idea for this, because I dont see a necessity to do it experimentally

Hi,

I was wondering if, during the pre-training of the LM alone, the sentences were prepended with a start symbol, just like they are during fine-tuning. If that's the case, could you please mention what is the name of the token in the learned vocab? If that's not the case, then wouldn't it introduce a bit of a mismatch wrt the pre-trained LM? Of course, the model is being fine-tuned so it will adapt, but then why would the start symbol be necessary for fine-tuning? Thanks!

does anyone know why the last one is excluded ?

Welcome to join this Wechat Group for discussion.

Dear guys,

I found that the position embeddings are concatenated with the word embeddings in the embedding layer.

My questions are:

1. During lm pretraining, did you mask out the logits from those position indices when computing the loss?
2. If I use the pretrained model as a LM to generate text, do I need to mask out these position indices' logits before softmax when sampling the next word?

BTW, I used the pytorch code ported by huggingface:
https://github.com/huggingface/pytorch-openai-transformer-lm
FYI, I also posted an issue there describing some details of my experiments:
huggingface/pytorch-openai-transformer-lm#36

• Da Xiao

was trying to use it as a language model to assign a score(could be PPL score) of a given sentence. Something like
P("He is go to school")=0.008
P("He is going to school")=0.08
Which is indicating that the probability of second sentence is higher than first sentence. Is there a way to get a score like this?

Thanks

hi

can anybody help me please?
how can i upload a huge amount of data for LM (for example about 2 G) ??

tnx

def rocstories(data_dir, n_train=1497, n_valid=374): storys, comps1, comps2, ys = _rocstories(os.path.join(data_dir, 'cloze_test_val__spring2016 - cloze_test_ALL_val.csv')) teX1, teX2, teX3, _ = _rocstories(os.path.join(data_dir, 'cloze_test_test__spring2016 - cloze_test_ALL_test.csv'))

in train.py, I get the following error:

File "train.py", line 201, in mgpu_train
for i, xs in enumerate(zip(*xs)):
TypeError: zip argument #1 must support iteration

I couldn't resolve this issue, any help would be really appreciated

Probably running ahead of things, but the generic model(s) you plan to release are for English only I suppose?

I would really be interested in ones for more languages, kind of like fasttext has done.

Otherwise I'd have to resort to training them myself on my single GPU :(

Any plans for more languages?

Hi,

Congratulations on the paper! Those of us who actually worked on ROCStories know how difficult it is!!

I have a small question on how embeddings are handled in the code.

we = tf.get_variable("we", [n_vocab+n_special+n_ctx, n_embd], initializer=tf.random_normal_initializer(stddev=0.02))
e = tf.gather(we, X)
h = tf.reduce_sum(e, 2)

I believe this is equivalent to embeddings_look_up() that people normally use...so we is word embedding. My question is: what is n_ctx (context embedding)? May I ask how is this used in the model?

Thank you very much!

Now that I looked at the code closer, is it an artifact of the Transformer Decoder??

I use the training data to train this model. (I make the wrong ending by random)
And use the test data to test.
The result is only about 60%. While the common embedding model can reach 65+%.
I'm not sure whether I use this model in a right way.
Do you ever try this ?

I see you use through the whole code 1-d convolutions, which technically should perform the same as Dense layers. The main difference I found on the internet was a shorter computing time for the Dense layer. Hence why I am wondering why you use 1D convolutions here?

Cheers

Hi,

I understand that there is a non-determinism due to GPU ops and I observed this as well when running twice the same code on the same GPU gave significant different results. However, I was wondering why the pytorch re-implementation https://github.com/huggingface/pytorch-openai-transformer-lm is actually giving the same results when running twice in a raw. Could it be that I am using a "wrong" version of TF? I have tensorflow-gpu 1.4.0, python 3.6, cuda 8.0 and cudnn 6.0. Thanks!

def get_ema_if_exists(v, gvs):
    name = v.name.split(':')[0]
    ema_name = name+'/ExponentialMovingAverage:0'
    ema_v = [v for v in gvs if v.name == ema_name]
    if len(ema_v) == 0:
        ema_v = [v]
    return ema_v[0]

def get_ema_vars(*vs):
    if tf.get_variable_scope().reuse:
        gvs = tf.global_variables()
        vs = [get_ema_if_exists(v, gvs) for v in vs]
    if len(vs) == 1:
        return vs[0]
    else:
        return vs

g, b = get_ema_vars(g, b) I think g, b is the original tensor, not the ema

This is very exciting work! Can you provide the steps to generate a language model using your architecture?

Thanks.

Hi!
I've noticed that the training code using 1d convolution with kernel size 1 in all invocations. Do we need convolution at all here? Why not replace it with the fully_connected layer?

Thank you for the great research and code! Regarding this section in the research paper:

"For CoLA (linguistic acceptability), examples are scored as the average token log-probability the
generative model assigns and predictions are made by thresholding. For SST-2 (sentiment analysis),
we append the token very to each example and restrict the language model’s output distribution to only
the words positive and negative and guess the token it assigns higher probability to as the prediction.
For RACE (question answering), we pick the answer the generative model assigns the highest average
token log-probability when conditioned on the document and question. For DPRD [46] (winograd
schemas), we replace the definite pronoun with the two possible referrents and predict the resolution
that the generative model assigns higher average token log-probability to the rest of the sequence
after the substitution."

I have tried to adapt the language model part of the model in the code to perform the above mentioned tasks. For instance, for CoLA, I fed in the encoded sentences and evaluated the results by thresholding the lm_losses output from the language model. However, the best Matthews correlation coefficient obtained is 0.015, far short of the 0.479 achieved in the paper. How exactly was the model configured to perform the task? Was it purely through the language model output, or was the supervised classification head used?

how could i train the language with chinese ，but the code only shows how to finetune the model. thanks

I find BERT uses BookCorpus (800M words) and Wikipedia (2500M words) but GPT only uses BookCorpus, even BERT has complex model structure which may leads to effect representation ability, the difference in evaluation result may also comes from training corpus. Have you ever try a bigger corpus like wikipedia?

The compare result also can imply BERT #task 2 influences.

what does the variable n_ctx mean ?
could you explain why it is computed like that ?
n_ctx = min(max(max(len(x[:max_len]) for x in X) for X in [trX, vaX]) + 3, args.n_ctx)

I trained this architecture on Japanese and got only 50% training accuracy. Is it normal here?

I suspect it is not 100% because we predict for entire sequence in this model, not only for n-gram as Word2Vec.

Thank you very much.

In your paper, the learning rate used in adam optimizer during pre-train is described as follows:
'We used the Adam optimization scheme [27] with a max learning rate of 2.5e-4. The learning rate was increased linearly from zero over the first 2000 updates and annealed to 0 using a cosine schedule.'
but what is the specific formula for this learning rate?

def get_ema_if_exists(v, gvs):      
    name = v.name.split(':')[0]  
    ema_name = name+'/ExponentialMovingAverage:0'  
    ema_v = [v for v in gvs if v.name == ema_name]  
    if len(ema_v) == 0:  
        ema_v = [v]  
    return ema_v[0]

Here is a piece of code in utils.py line 117 - 123
When you use the list comprehension, the variable v overwrites the argument v in the function.
So what returned is always the last tensor in gvs if the length of ema_v is 0.

This may cause problem. In this case, you are using the same variable for all normalization function.

Have you released the code for language model training?

In the code that does the convolution, there is a separate implementation for convolution with filter width of 1 and convolution with other filter widths.

Convolution with a filter width of 1 is special cased due to a faster implementation and it is the only filter width used by the current code. Other implementation has a comment #was used to train LM.

Does it mean that a different convolution filter width was used for language modeling or was it just using a less efficient implementation but with the same filter width?

Hey, it seems from the code that the position embedding matrix W_p was not used. Am I correct?

h_0 = UW_e +W_p 
h_l = transformer_block(h_{l−1})∀i ∈ [1, n]
P(u) = softmax(h_n W_e^T )

Thank you.

X_train = tf.placeholder(tf.int32, [n_batch_train, 2, n_ctx, 2])
xmb[:, :, :, 1] = np.arange(n_vocab+n_special, n_vocab+n_special+n_ctx)
why there is a channel of additional tokens?

Maybe I don't interpret the model(...)-function correctly, but I see the following:

While training you put the correct and wrong rocstories together into the decoder. They both go through the embedding + decoder and then into the sparse_softmax_cross_entropy-function.

This means, though, that the model also learns to generate wrong sentences, or am I missing something?

My intuition would be to set all masks to 0 for the wrong sentences?!

Thanks and regards

Hello,

First, I would like to thank the authors of this paper for releasing their source code.

Is there a plan to use the same approach using a Universal Transformer as base architecture? Would the adaptive computation time (ACT) mechanism transfer to other tasks?

And more importantly, if this new transformer can be used, do you think the gain would be noticeable?

I ran quora_similarity.py, and didn't modify any codes, then I got 0.77 accuracy for 0.4 class balance, any ideas for this? Thanks.

Hi,

I tried to run this code to reproduce the accuracy of 85%. But I have executed 3 times and got only 53% accuracy each time.
Could you please tell me some tips about the instructions to run this code or set parameters?
I did't change any code and used the command python train.py --dataset rocstories --desc rocstories --submit --analysis --data_dir [path to data here] in readme.

Thanks

Hello. Sorry, but i can't understand how this function work. In my tests in most cases the result is equal to original token parameter value.
https://github.com/openai/finetune-transformer-lm/blob/master/text_utils.py#L49

could u share the pretrain code?

Is it possible? To achieve maybe 90% of the experiment result in the paper.
Thank you very much!

Hi,

We tried several settings based on your code and paper, but unfortunately we cannot reproduce the RACE score (the training loss decreases, but the dev accuracy only reaches 0.26).
Could you tell me some tips about the parameter or code modification for achieving the performance?

Thanks

Thanks for the code release. Do you have any timeline to release the code to train the LM and finetune on the other 11 tasks?