Break down into smaller issues and then assign

Step 1: Find Sentiment data for English tweets
Step 2: Translate that to Hindi using something like Textblob
Step 3: Transliterate that Hindi to Roman Script: https://github.com/libindic/indic-trans

Hello guys,
I am trying to know how to annotate Hinglish data that should be useful for finetune language models or building a language model. I saw your train.json file on that text is tweet, and clean_text is preprocessed text. And what about Hindi words in the tweet? Are Hindi words tagged? I mean, I read in an article they were annotating Hinglish tweets with corresponding English version sentences. In another article, they were mentioned like each word in Hinglish sentence tagged with corresponding language and whole sentence tagged with either sentiment classification labels or any other classification problem labels.

I just wanted to know what process we were using for training data annotation purposes. Can you please tell me?

• Without confidence learning
• With confidence learning

1. Integrate with this repo if needed
2. Make sure we both have access to the WanDB dashboard via Github accounts

https://www.wandb.com/

We will use these results to manually verify how much to trust this dataset labels itself

E.g. if the error is say, more than 5% - we will pass this value during test time prediction, which will have the same label error rate hopefully

The people who preprocessed this data, don't know how to use regex to remove URLs completely.

They just removed the special characters, without removing the URL itself: https t co tsrsbu

• Use a randomized 5-fold CV, with a fixed seed state instead of a test split
• Measure F1, and Accuracy - just accuracy has no meaning
• Always print the confusion matrix via sklearn, or go all the way and seaborn it 🥇

Given above, would it be more useful to this within the notebook instead of a separate wonky CleanTwitter class?

They have included Filipino tweets in a dataset of Hinglish tweets 🤦‍♂

I wonder how did they assign it a sentiment - do they speak Filipino too?

The truncated tweets also indicate that this was longer 280 char tweet, which they truncated to 140 characters.

Example:

happy birthday seatmate thank you kasi masginanahan ako magaral kasi katabi ko kayo ni dom hahahhah thank you are https t co jyeeskiyf

which translates to:

happy birthday seatmate thank you for helping me study because i was with you hahahhah thank you are https t co jyeeskiyf

Reference Code: https://github.com/NirantK/Hinglish/blob/2798fda87b9b28fa1d7921203ed466c9fd23a28d/hinglishutils.py#L401-#L408

Recommended approach from Huggingface from here:

batch_sentences = ["Hello I'm a single sentence",
                   "And another sentence",
                   "And the very very last one"]
encoded_inputs = tokenizer(batch_sentences)
print(encoded_inputs)

After #33-#35 are done