Preprocessor step disposing numbers in (variable) names about code2vec HOT 4 CLOSED

Hi @daveymathijssen ,
Thank you for your interest in our work!

The C# constructor was contributed by researchers from Microsoft, so I am not sure why did they remove numbers.
I have might removed number in the JavaExtractor as well, I'm not sure.
But if it improves your metrics when you do not remove the numbers, that's great!

Best,
Uri

from code2vec.

HI @urialon,
Thanks for your answer!

In the JavaExtractor, the same is happening in the normalizeName method.
Do you have any clue why?

from code2vec.

I'm guessing that at the time,
We did not want to spend embeddings on numbers, as we had ~1M embedding vocabulary anyway.
This is solved in newer models by segmenting tokens into subwords.

You can also check out our newer models

• code2seq: https://code2seq.org/
• PolyCoder: https://github.com/VHellendoorn/Code-LMs
• CodeBERT that we finetuned on multiple languages: https://github.com/neulab/code-bert-score#backend-model

Best,
Uri

from code2vec.

Thank you for your fast responses and insights!

from code2vec.