graykode / xlnet-pytorch Goto Github PK

Hi author,

Any idea why the performance accuracy is so low on the sample task you provide?

I added the following lines to test accuracy:
output = logits.transpose(1, 2)
predicts = F.softmax(output, dim=1).argmax(dim=1)
print((predicts == target).sum().item() / len(target))

The accuracy for the task is ~0.01% after 100 epochs.

Hi

thanks for sharing this; I have a quick question , do we need to create data for every epoch?

First the error — I get this both when trying to run the notebook locally (ubuntu 18.04) and from Colab:

Traceback (most recent call last):
  File "main.py", line 89, in <module>
    num_predict=args.num_predict)
  File "/content/xlnet-Pytorch/data_utils.py", line 345, in make_permute
    reuse_len)
  File "/content/xlnet-Pytorch/data_utils.py", line 292, in _local_perm
    non_mask_tokens = (~is_masked) & non_func_tokens
RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'other' in call to _th_and

Any ideas?

For the general question; I really want to pretrain from scratch with my own small corpus. Any tips on how I might go about doing that?

Thanks

I want to do sentence embedding using your pytorch code. but i do not find how to make test data input and inference code

As mentioned above.

If I run the code with default arguments (and use data.txt from the repository) I get the following message:

Traceback (most recent call last):
  File "C:/Users/matej/git/xlnet-Pytorch/main.py", line 89, in <module>
    num_predict=args.num_predict)
  File "C:\Users\matej\git\xlnet-Pytorch\data_utils.py", line 345, in make_permute
    reuse_len)
  File "C:\Users\matej\git\xlnet-Pytorch\data_utils.py", line 292, in _local_perm
    non_mask_tokens = (~is_masked) & non_func_tokens
RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'other'

I use Python 3.6.9 and PyTorch 1.2.0.

In your implementation, the FFN module only has one linear layer. is it a bug?

Hello authors,

Thank you for your re-implementation. I look forward to using it. I just wanted to confirm your reimplementation results because I did not see any in the README/Colab.

Have you reproduced the XLNet results (any of Squad/IMDB/GLUE/etc.) by finetuning with PyTorch?

can you help me ?
D:\xlnet-Pytorch-master>python main.py
Model name 'bert-large-uncased' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
File "main.py", line 58, in
model = xlnet.XLNet(n_token=len(sp.vocab), n_layer=6, n_head=4, d_head=8,
AttributeError: 'NoneType' object has no attribute 'vocab'

It seems that the parameter initialized with randn (

Usage code runs successfully, but I can't find a saved parameters.

@graykode Can you explain how batch training would be conducted? For example, what if we had multiple input files for training data?

Currently, training is done using only a single data file. For multiple data files, would data_utils._create_data have to return a batch of features?

The permutation function " _local_perm" is confused.

The generated permutation mask is not consistent with what the original paper reported for the permutation order "3→2→4→1" in Figure 2.

When I run the code on the colab, I got the above error. I wonder where did I do wrong and what is the suitable codes environment requirements. Thank you very much!

Traceback (most recent call last):
File "main.py", line 89, in
num_predict=args.num_predict)
File "/home/hemengge/xlnet-pytorch/data_utils.py", line 335, in make_permute
is_masked = torch.Tensor(feature.pop("is_masked"))

TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8

The default setting is to use the bidirectional data, attn_type='bi', but bsz=1.
But in this function,

if mem is None , men_len = 0, have some bugs