Comments (12)
Hi @ts-thomas, last year you said:
The next main release will get an improvement of handling all language-specific features.
As FlexSearch is currently in 0.7.0 release candidate, I was wondering if there is some new feature about languages processing.
I recently created a project called SearchinGhost, it is an in-browser search plugins for Ghost CMS powered by FlexSearch. I am really happy with FlexSearch (thank your for the work done!) but I would like to bring multi-lang capabilities. For now, with what I read in the issues, I came up with these language default options:
Latin:
FlexSearch.create({
encode: "simple",
tokenize: "forward"
});
Arabic
FlexSearch.create({
encode: false,
rtl: true,
split: /\s+/,
tokenize: "forward"
});
Cyrilic, indian (any word separated by space language)
FlexSearch.create({
encode: false,
split: /\s+/,
tokenize: "forward"
});
Chinese, Japanese or Korean (with dedicated chars w/o spaces)
FlexSearch.create({
encode: false,
tokenize: function(str){
return str.replace(/[\x00-\x7F]/g, "").split("");
}
});
Do you think there is any possible improvement/optimisation?
EDIT: I finally found this relevent documentation about the v0.7.0 - https://github.com/nextapps-de/flexsearch/blob/0.7.0/doc/0.7.0.md. Hope this version will be out one day :)
from flexsearch.
Also take into account to use the "rtl" option for right-to-left support:
var index = FlexSearch.create({
encode: false,
rtl: true,
split: /\s+/,
tokenize: "forward"
});
from flexsearch.
@mahnunchik The latest version v0.6.1 has the new option split when creating an index. This makes it possible to use built-in tokenizer like "strict" or "forward" (which is required by the contextual index) and also handle word splitting separately.
var index = FlexSearch.create({
encode: false,
split: /\s+/,
tokenize: "reverse"
});
index.add(0, "Фообар");
var results = index.search("Фообар");
var results = index.search("бар");
var results = index.search("Фоо");
from flexsearch.
This one works too:
const index = new FlexSearch({
tokenize: function(str){
return str.split("");
}
});
from flexsearch.
@ts-thomas thank you! It helps me in my task.
Maybe there is sense to use more common split
by default? To cover all languages which use space as separator of words.
It will be hard to find root of the problem for string like foobar αβγ
with default split option /\W+/
.
from flexsearch.
The next main release will get an improvement of handling all language-specific features. This improvement will also cover splitting words.
from flexsearch.
The mentioned solutions here helped me to make FlexSearch indexing Arabic text. Thanks guys.
from flexsearch.
Hello. Try this settings: https://github.com/nextapps-de/flexsearch#cjk-word-break-chinese-japanese-korean
var index = FlexSearch.create({
encode: false,
tokenize: function(str){
return str.replace(/[\x00-\x7F]/g, "").split("");
}
});
index.add(1, "Фообар");
var results = index.search("Фообар");
When you just want to index whole words (not partials) then this may be better:
var index = FlexSearch.create({
encode: false,
tokenize: function(str){
return str.split(/\s+/);
}
});
from flexsearch.
Hello @ts-thomas
Yep, it works with the following options.
const index = new FlexSearch({
tokenize: function(str){
return str.replace(/[\x00-\x7F]/g, "").split("");
}
});
But why it doesn't work with the forward
tokenizer?
from flexsearch.
The forward tokenizer splits words via regex, but this regex will also remove cyrillic chars actually. The tokenizer needs to be enhanced to use „forward“ with non-latin content.
from flexsearch.
It would be really helpful 😉
from flexsearch.
For some reason, the solution to this in typescript seems to be different. I'm just posting how I've gotten it to work in hopes it will help others.
Note: I'm using "flexsearch": "0.7.11"
because without that I was facing this issue while using typescript.
I was not able to follow the FlexSearch.create(< options >)
nor the new FlexSearch(< options >)
paradigms I saw other's using here due to compile errors. The split
field doesn't exist on the new Index(< options >)
's options type so that also caused compile errors. I tried to specify the charset as "cyrillic" and "cyrillic:default" but that also wasn't working.
I found that specifying the encode
to a function of your choice is all you need to do.
It seems that the only function that actually matters is the encode function. I looked at the source and I played a bit and it seemed that when you specify an encoder, the tokenize, stemmer, and charset it doesn't affect anything.
Here's my working code:
import { Index } from "flexsearch";
import { PorterStemmerRu, WordTokenizer } from "natural";
const tokenizer = new WordTokenizer();
const index_ru = new Index({
encode: (str) => {
let ret = tokenizer.tokenize(str);
for (let i = 0; i < ret.length; i++) {
ret[i] = PorterStemmerRu.stem(ret[i]);
}
return ret;
},
});
I'm using the natural library to handle the stemming.
I'm new to this library so please correct any horrible mistakes I've made!
from flexsearch.
Related Issues (20)
- RangeError: Invalid string length HOT 2
- How to import Document in nodejs with ESNext modules (typescript)?
- Invalid regular expression
- Suggestion and tokenize "forward"
- Webpack cannot resolve flexsearch HOT 1
- `IndexOptions` TS interface is missing some options HOT 1
- Current NPM package is stale HOT 2
- Cloning of flexsearch objects
- new Document causes `.default is not a constructor` error HOT 1
- Benchmark link is broken HOT 1
- Setting a string as "encode" doesn't work (confusing document?)
- How to return context (+ or - few lines) around the hits of given search text
- Enriched document search showing duplicate results
- I have a question about how the results are sorted
- Do not force string in Index#add() when custom encoder
- "Document Indexes" link in readme is broken
- Typo in cdn link
- TypeScript doesn't allow omitting id from the document descriptor while the README says it is allowed
- document search option "pluck" is rejected by TypeScript
- Some documents may appear multiple times in the search result
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flexsearch.