The repository contains a library for classifying programming and markup languages, submitted to the second round of the Telegram ML Competition 2023.
-
Preprocessing
- Tokenize the input string with the following regular expression:
(\b[A-Za-z_]\w*\b|[!#$%&*+:\-./<=>?@\\^_|~]+|[ \t(),;{}\[\]`"'])
- Using a predefined vocabulary, calculate a TF-IDF value for each known token
- Tokenize the input string with the following regular expression:
-
Prediction
- The random forest model was used for classification
- Each decision tree was ported to C code consisting of
if/else
statements
- 1000 TF-IDF features
- 50 decision trees
- maximum tree depth of 50
- Accuracy:
94.55%
- F1:
94.77%
- Inference time:
3.95 ms
Keep in mind that these metrics were measured on a similar dataset to the original training dataset and may differ significantly when measured on different data. For more details about the trained classifier, the confusion matrix can be found in the notebook.
Source | Samples |
---|---|
Telegram | 21738 |
RosettaCode | 19820 |
GitHub | 106836 |
Generated | 30165 |
Manual | 526 |
- Telegram: code snippets from the first round of the competition (available labels are
CODE
andOTHER
) - RosettaCode: code snippets from https://rosettacode.org/ (missing languages
are
CSS
,DOCKER
,FUNC
,HTML
,NGINX
,OTHER
,SOLIDITY
,TL
andXML
) - GitHub: code snippets from GitHub using a modification of the original
Kaggle Notebook
based on GitHub Repos dataset (
TL
andFUNC
languages are excluded). - Generated: code snippets generated via ChatGPT by tzador for the first round of the competition.
- Manual:
TL
andFUNC
snippets were gathered semi-manually from GitHub.
The combined training dataset has an unbalanced class distribution with a predominance of the OTHER
class.
More information about the collected data can be found in the notebook.
To export a shared library libtglang.so
built for Debian GNU/Linux 10 (buster) run the following command:
DOCKER_BUILDKIT=1 docker build --no-cache --target export-lib --output out .
NOTE: Due to the generated C files with large nested if/else statements, the compilation stage may take approximately 15 minutes.
To evaluate metrics of the built library on a test dataset, run:
docker run --rm -it $(docker build -q .)