soumendrak / openodia Goto Github PK

The current release action is based on an official action implementation for semrel and is buggy when triggered.

Is your feature request related to a problem? Please describe.
Add the English to Odia dictionary corpus into the library.

Describe the solution you'd like
The dictionary can be found in this Kaggle link.

Describe alternatives you've considered
No alternatives yet.

Additional context

• We can further improve it by adding parallel sentences pairs corpus from MTEnglish2Odia repo.
• The corpus position can be inside a corpus folder in openodia and we will import that in the __init__.py file.
• Its signature should be like

from openodia import en2or_dictionary

Warning Dump from the CI job-

"Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: codecov/[email protected]. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/."

"Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/setup-python@v2, actions/upload-artifact@v2. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/."

Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: actions/checkout@v2, github/codeql-action/init@v1, github/codeql-action/autobuild@v1, github/codeql-action/analyze@v1. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/

Is your feature request related to a problem? Please describe.

• Stemming is the process of reducing the inflected words to their root form.

Describe the solution you'd like

• In Odia, there are many letters/suffixes/prefixes which can be removed to bring words to their root form and reduce variance.
• Few of these are like:

"ଉଛ",
"ଉଛି",
"ଉଥିଲା",
"ଉଥିବ",
"ଉଥିବି",
"ଅଛ",
"ଅଛନ୍ତି"

• After doing word tokenization rstrip these letters from the tokens.

Reference

• Table-4 of this paper

tokenizing words removing punctuations from the text.

Expected:

>>> from openodia import ud
>>> ud.word_tokenizer("ଭାରତୀୟ ସର୍ବୋଚ୍ଚ ନ୍ୟାୟାଳୟ, ଭାରତର ଉଚ୍ଚତମ ନ୍ୟାୟିକ ଅନୁଷ୍ଠାନ ଅଟେ ।")
[
"ଭାରତୀୟ", 
"ସର୍ବୋଚ୍ଚ", 
"ନ୍ୟାୟାଳୟ", 
",", 
"ଭାରତର", 
"ଉଚ୍ଚତମ", 
"ନ୍ୟାୟିକ", 
"ଅନୁଷ୍ଠାନ", 
"ଅଟେ",
"।"
]

However, currently, comma and purnnachheda symbols are not coming in tokenized text:

Expected:

>>> from openodia import ud
>>> ud.word_tokenizer("ଭାରତୀୟ ସର୍ବୋଚ୍ଚ ନ୍ୟାୟାଳୟ, ଭାରତର ଉଚ୍ଚତମ ନ୍ୟାୟିକ ଅନୁଷ୍ଠାନ ଅଟେ ।")
[
"ଭାରତୀୟ", 
"ସର୍ବୋଚ୍ଚ", 
"ନ୍ୟାୟାଳୟ", 
"ଭାରତର", 
"ଉଚ୍ଚତମ", 
"ନ୍ୟାୟିକ", 
"ଅନୁଷ୍ଠାନ", 
"ଅଟେ"
]

Difference:

[
"ଭାରତୀୟ", 
"ସର୍ବୋଚ୍ଚ", 
"ନ୍ୟାୟାଳୟ", 
- ",", 
"ଭାରତର", 
"ଉଚ୍ଚତମ", 
"ନ୍ୟାୟିକ", 
"ଅନୁଷ୍ଠାନ", 
"ଅଟେ",
- "।"
]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

• Currently, while translating we hit Google translate API directly.

Describe the solution you'd like
A clear and concise description of what you want to happen.

• We should try as much as possible to reduce latency by hitting the offline dictionary first. If not found there then go to hit the Google Translate API.
• In this way, this will act as a cache.
• This has a dependency on #6

Describe alternatives you've considered

• Open for alternatives.

I wrote a Python code using openodia library.
When I try to convert "Three Hundred and Twenty One" using other_lang_to_odia(), it is returning "Three Hundred and Twenty" only - One is missed, and I have no idea why it is ignored.
Any clues ?