Comments (5)
I can implement the following.
The scenario is like this: First, configure a mapping with languages you want to detect in languages
. Then, configure other fields where to map the text of successfully detected languages to, in a parameter language_to
.
{
"someType" : {
"properties" : {
"someField":{
"type" : "langdetect",
"languages" : [ "de", "en", "fr", "nl", "it" ],
"language_to" : {
"de": "german_field",
"en": "english_field"
}
},
"german_field" : {
"analyzer" : "german",
"type": "string"
},
"english_field" : {
"analyzer" : "english",
"type" : "string"
}
}
}
}
In this example, submitting a text "This is a small example of english text"
to someField
will index en
to thesomeField
field with langdetect
type, but also the text will be passed to the field english_field
. German text would be indexed into field german_field
, using a different analyzer.
It is up to the user to configure the field analyzers and the language_to
mapping appropriately. There are cases where detected languages don't have a Lucene language analyzer. So it is not possible to implement a total automatic scenario, covering all languages that can be detected, and covering all Lucene language analyzers.
Another issue is indexing multilanguage text into a single field. Here I recommend the ICU analyzer. ICU can apply normalization / folding / tokenization based on Unicode scripts which is the best method to search for multilanguage in a single field. Stemming is not applied.
from elasticsearch-langdetect.
Hi,
I see so many requests for auto-language detection and use right analyzer for indexing. Is this issue currently being tracked ? Any response is really appreciated.
Regards
from elasticsearch-langdetect.
With the removal of _analyzer being specified in the query (in elastic/elasticsearch#9279), auto selection of the analyzer for a field doesn't really make sense as far as I can tell. Each field has only a single analyzer associated with it, so you can't really analyze on the fly based on lang detect.
So either you are putting your content into a field that is agnostic about the analyzer and doing to lang detection to filter on, or you make one call to determine the language of your content, and then index your data to the appropriate field for the appropriate analyzer.
So for instance we have separate fields like:
- content.en
- content.es
- content.fr
- etc...
- content.default
from elasticsearch-langdetect.
Released version 2.4.4.1 with the language_to
feature.
from elasticsearch-langdetect.
@jprante for language detection can you provide a default/fallback language_to
when threshold is below some confidence level? (which probably would point at an analyzer that is somewhat language neutral with ICU that is safer across a lot of languages but not perfect).
from elasticsearch-langdetect.
Related Issues (20)
- Seems like landdetect 5.3 does not work or documentation has incorrect examples HOT 6
- problem with decoding escaped unicode string HOT 2
- Error installing 5.3.0.2 plugin on ES 5.3 HOT 2
- Detection problem with Unicode escaped JSON HOT 2
- Am I using the short-profile correctly? current example outdated HOT 1
- ETA for langdetect for ES 5.3.2 HOT 1
- REST API returns an application/yaml response
- ES 5.4 HOT 3
- Is it possible to update mapping for "langdetect" type?
- Can't store langdetect field ? HOT 1
- Can't aggregate ? HOT 2
- plugin [langdetect] is incompatible with version [5.4.1]; was designed for version [5.4.0]
- ES 5.5
- High memory usage with langdetect
- Search language not found HOT 1
- How to set config in elasticsearch.yml
- ES 5.6 support
- mapper_parsing_exception for some languages
- Cannot start elasticsearch when change language file
- ES 6.x support for language detection? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch-langdetect.