Comments (10)
When reviewing the code to adapt it for implementation in Portuguese, I realized that opting for integration in the .csv file would not make much sense. In my opinion, exploring an alternative implementation method has numerous potential advantages and benefits. Here's a breakdown of the key points:
Benefits:
-
Efficiency: By splitting the training data by language, the code avoids retraining the entire model when a new language is added or an existing one is updated. This saves time and computational resources.
-
Scalability: The optional langs parameter allows specifying which languages to check for profanity. This can improve server performance by avoiding unnecessary checks for languages not in use.
-
Maintainability: Separating the training data by language makes the codebase more manageable. It simplifies adding new languages and avoids cluttering the code with a massive combined dataset.
from profanity.dev.
Great discussion. From a user perspective I would find it confusing if there were multiple variations of en
and eng
, because intuitively there might not be a difference. By the way, great question of if this should be in the same database or namespace. The benefit would be that it's much simpler to set up.
On the other side, the risk might be that a word in spanish looking similar to an english swear word might get flagged without meaning something bad in spanish. Then again, some profanities remain the same, i.e. the most popular english profanities also work in spanish I suppose - we'd have to re-index all of them for the spanish, portugese, etc. versions. So I propose just adding them to the same database and seeing how that goes, the lang parameters make sense
from profanity.dev.
Was going to drop some Chinese and Thai in and submit a PR, but saw this issue first and I couldn't agree more.
Having a langs param would be really important for getting rid of false positives, especially regarding languages written using something other than the Latin alphabet when they are romanized or abbreviated.
from profanity.dev.
I have little to no experience with lanague-based content, but I've seen that the ISO 639 is the go-to web/API standard. Some resources:
- https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes
- https://standards.iso.org/iso/639/ed-2/en/Access%20to%20the%20databases%20of%20the%20ISO%20639%20Language%20Code.pdf
- https://www.loc.gov/standards/iso639-2/php/code_list.php
Here are two sub-issues:
- do profanity words change across individual 3-letter languages (eg Southern
sou
and Northern Thainod
)? 1 - do profanity words change across local variation (eg US
en-US
vs UKen-UK
, or Portogual vs Brazil...)? 2
Although I'm pretty sure the answer is no for both questions in most cases, I would like to implement the ability to use both 3 letter codes and localization too.
If we were to implement 3 letter codes and localization:
We could have a training_data
folder with an en
subfolder, which would contain en.csv
as the main "index" for the macro language, eng.csv
as an extension to the relative macro language (ie en
) and then an en-US.csv
for any local extension to the 2 letter code language or eng-US
for even finer localisation (probably it's never going to be used this last option tbh).
Params
Here is my idea:
If lang=en
is passed then only en
is used
If lang=eng
is passed then both en
and eng
are used
If lang=en-US
is passed then both en
and en-US
are used
Since all cav are in the 2 letter code folder we have a super easz way to know the 2 letter code from the 3 letter one.
Langs endpoint
Also as a bonus, a simple /api/langs/
could return the supported language list.
Footnotes
-
Difference between 2 letter and 3 letter codes https://en.wikipedia.org/wiki/ISO_639-3 ↩
-
Country codes: https://www.iso.org/obp/ui/#search ↩
from profanity.dev.
On the other side, the risk might be that a word in spanish looking similar to an english swear word might get flagged without meaning something bad in spanish. Then again, some profanities remain the same, i.e. the most popular english profanities also work in spanish I suppose - we'd have to re-index all of them for the spanish, portugese, etc. versions. So I propose just adding them to the same database and seeing how that goes, the lang parameters make sense
An example: the word "negro" in Italian is literally the n-word, however in Spanish it's just the black color, without any negative connotation whatsoever (as far as I know)
from profanity.dev.
however in Spanish it's just the black color, without any negative connotation whatsoever (as far as I know)
Yes, same thing in Portuguese.
from profanity.dev.
From a user perspective I would find it confusing if there were multiple variations of
en
andeng
, because intuitively there might not be a difference. By the way, great question of if this should be in the same database or namespace. The benefit would be that it's much simpler to set up.
It would be nice to have some input on this from the community on Asian and Arabic languages (on Wikipedia I saw multiple 3-letter languages under the "ar" common macro one)
Some profanities remain the same, i.e. the most popular english profanities also work in spanish I suppose - we'd have to re-index all of them for the spanish, portugese, etc. versions.
I think an easier approach would be to just let the dev pass the eng
parameter if they're concerned with it.
For English/German-root-related languages there are certain shared words (for example English-German with "shit" and probably more that you @joschan21 know better), but at least I can tell you that most profanities in Italian are... In Italian. Like, you might hear someone say the n-word in English, however it's not extremely common.
Some English words are used as neologism, profanity or not, but I wouldn't put them in other databases.
With this said, Italian is full of swear words, so it might also just be we don't "need" English for profanities.
Also, I'm sorry to ask, but: what's the difference between "namespace" and database"? Is the database the single training csv data?
from profanity.dev.
I tried writing harsh words in Indonesian but it wasn't detected that they were toxic
from profanity.dev.
Quick update: namespace support was added last week to upstash js api: upstash/vector-js#25
I'm testing some stuff with it :)
Edit: answering my old question: namespaces are a way to group data under a single index in a similar fashion to metadata, however, contrary to metadata, it is selectable on query. In other words: one database with groups.
from profanity.dev.
Also @joschan21 which model do you recommend to convert the raw text to vector data? I'm writing a guide in a README.md to keep track of what I did in order to get started and would like to know if you have a recommendation.
from profanity.dev.
Related Issues (20)
- word "suck" is flagged as highest profanity HOT 2
- Feature Request: Add PDF File Input and API Support HOT 2
- The letter f is flagged for profanity higher than 0.850
- The letter "f" is flagged for profanity higher than 0.850 HOT 1
- Support for text symbol HOT 2
- Add a README.md file at root HOT 4
- feat: add portuguese language support HOT 1
- Python library HOT 1
- Creative slurs to consider HOT 2
- Add semicolons to end of line HOT 1
- Just some tests HOT 2
- [SUGGESTION] Allow configuration HOT 2
- Repeating Consecutive Words causes False Positives HOT 1
- here are some workarounds that should be worked on! HOT 2
- the formatted data getting undefined after running the for loop functiion
- "don't waste oxygen, just do it" is not detected HOT 1
- pick a license
- some words getting detected for no reason HOT 4
- The API is down and responds back with an error. HOT 1
- Normalise score
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from profanity.dev.