pylint has a number of failures on a number of files. To enable it in the actions we need to fix all of the pylint errors throughout the files first. After this is done the "pylint" line in the pythonapp.yml file can be uncommented to run again.

Perform research on how we could add methods to HappyBERT that enable fine tuning. Post a 200 word paragraph to the report section of our google drive about your findings. Include potential resources we could use.

Hi Eric,

Thanks for your wonderful work! I was wondering you would like to integrate some other recent models like ALBERT in the near future. That would make the experiments using happy-transformer more comprehensive!

Use the outputs of "k option in predict_mask" as features for a final model that optimally combines these outputs.

Do research on the structure of the model including specifics on the features and the output. It would be better if the model could predict k options as well instead of only the top 1.

Thanks!

Make sure you can get GPU support to work. Read their documentation and include all of the available inputs.

The finetuning method for language models returns the finetuned model.

Need a way of going from base model to happytransformer model.
model.from_pretrained() requires a directory but we only have a model object.

PR #18 created the yml file manually. However, should use the github actions tab to create the workflows

Good starting point: https://github.com/github/gitignore/blob/master/Python.gitignore

Perform research on how we could add methods to HappyBERT that enable fine tuning. Post a 200 word paragraph to the report section of our google drive about your findings. Include potential resources we could use.

All other language models do not perform as well as HappyROBERTA large for masked word prediction. We should encourage users to use HappyROBERTA Large by displaying a logger message if they use a suboptimal language model. This message will encourage them to use HappyROBERTA Large.

There are still some situations where a user may want to use another model, so we will keep them available.

Hi I tried to run the example script on Readme.md and it gives me the error that XLNet-base-uncased not found

Model name 'xlnet-base-uncased' was not found in tokenizers model name list (xlnet-base-cased, xlnet-large-cased). We assumed 'xlnet-base-uncased' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.

Thank you

Lots of code is reused amongst the child classes for predict_mask. Create methods within the parent class that perform some of the common tasks to make the code within the child classes shorter and easier to understand.

Hi!

Thanks very much for the great work! I found a trouble of ROBERTA's masked word prediction results are not the same as the official fairseq one.

You can test it by the example here: https://github.com/pytorch/fairseq/tree/master/examples/roberta#filling-masks

Will try it out + leave a review within next 2 weeks

Make sure you can get GPU support to work. Read their documentation and include all of the available inputs.

Write a 200 word summary of your findings and post it to the report section of the google drive. Focus on what modules need to be added to what we currently have.

Use "predict_mask_with_options" for each child class to generate results for the WSC273. Within the WSCTesting class, create a method that performs the test for each child class. Also create a single method that calls each of these tests at once and outputs the results.

Hello from Russia! Can you add language selection support? Accordingly, Russian. I have the task of predicting the next word from an available list of words. Or, if there is, how can I use it?

Hi,

We were running the fine-tuning mwp examples and found it reported a bug saying "Target -1 is out of bound". After debugging, we find that the doc of BertForMaskedLM says

masked_lm_labels (:obj:torch.LongTensor of shape :obj:(batch_size, sequence_length), optional, defaults to :obj:None):
Labels for computing the masked language modeling loss.
Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring)
Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels

Thus, we modified the mask_tokens function by labels[~masked_indices] = -100 # We only compute loss on masked tokens. And it works now. Not sure if it is correct, could you please do a double check ?

Create a sentence perplexity method for BERT. Perhaps this method can be placed in the parent class; that way, all children can use it.

Make sure you can get gpu support to work. Read their documentation and include all of the available inputs.

Create a transformer method to predict k number of masked words and return a list with the options in descending order by softmax

You will need to create special cases for models like XLNet that only have cased models.

Is there a way to predict multiple masked words for this? So for example if I have a sentence:

"[MASK] have a [MASK] dog and I love [MASK] so much"

Thank you so much!

Make sure you can get GPU support to work. Read their documentation and include all of the available inputs.

    def __init__(self, tokenizer, file_path, block_size=512):
        assert os.path.isfile(file_path)
        with open(file_path, encoding="utf-8") as f:
            text = f.read()

        tokenized_text = tokenizer.encode(
            text, add_special_tokens=True)  # Get ids from text
        self.examples = []
        # Truncate examples to a max blocksize
        for i in range(0, len(tokenized_text) - block_size + 1, block_size):
            self.examples.append(tokenized_text[i:i + block_size])

Here it seems to separate the file in file_path with the block_size instead of the \n ? Thus, the provided example in the readme cannot be trained.

Hi,

I found that the fine-tuned model did not do better on the mwp on the trained corpus, so I debug the code by showing its learning rater as follows in the train function.

  if global_step % logging_steps == 0:
                # tb_writer.add_scalar("lr", scheduler.get_lr()[0], global_step)
                print()
                print("\t lr:", scheduler.get_lr()[0])
                print("\t avg loss:", (tr_loss - logging_loss) / logging_steps, global_step)
                logging_loss = tr_loss

However, I found that lr is always 0..

Hi Eric,

Thanks for the feature of fine-tuning MWP. I am wondering if we can write our own customized masking strategy instead of random sampling?

Simply put, I have a list of sentences where masks are already created, and we know the associated words for each mask. Can we fine-tune HappyBERT/HappyROBERTA for them?

Thanks!

Include the purpose of HappyTransformer. Mention our future goals. Finally, write documentation on how to create child classes and use the methods we have completed thus far.

In predict mask and next sentence prediction there is no us of the "to(self.gpu_supprt)" to send the model (and possibly tensors) to the gpu. Investigate if this is correct and if thise needs to be added

Modify the code Eric produced during the Microsoft Hackference to use HappyRoBERTa

Use one of the various continuous integration (CI) tools to report the build status and test coverage (when there are tests) of the project.

travis CI seems like a quick and easy option.

I uploaded finetuning.ipynb on my branch the error I keep getting is RuntimeError: CUDA error: device-side assert triggered.

I need someone experience to debug this bad boy.

When options are given and a num_results is specified only the first "num_results" of the options are being returned, even if those are not the highest scored options

Create a predict_mask_with_options method for RoBERTa. If the Fairseq libray does not provide enough functionality to accomplish the task, then convert the RoBERTa class to the standard Transformer library implementation.

Github actions needs a pythonapp.yaml file to define the workflow.
Also needs a requirements.txt to run the tests