Comments (2)
Thanks for your suggestion! We're aware of the sparse index here and have some prototype at #552. For the weighted keyword search, we have tried https://github.com/tensorchord/pg_bestmatch.rs. Does it work for you? We're also thinking of a better way to directly build text index for bm25 search
from pgvecto.rs.
Thanks for aswering, i will take a look at pg_bestmatch.rs
.
I think that part of storing sparse vectors are solved. However i think, the query part must be rethinked.
I honestly think that the native to_tsquery
API (link1 link2) is great and can be replicated for sparse vectors (maybe to_svector_query
). The to_tsquery
API offers great flexibiliy (boolean operators):
SELECT to_tsquery('english', 'The & Fat & Rats');
SELECT to_tsquery('english', 'Fat | Rats');
SELECT to_tsquery('simple', 'Fat | Rats');
I think this type of API is great and can solve the previous animal keywors example, and also instead of 'english' we can add custom sparse encoders (TFIDF, BM25, SPLADE, CUSTOM)
SELECT to_svector_query('tfidf', 'We begin, as always, with the text.');
SELECT to_svector_query('bm25', 'We begin, as always, with the text.');
SELECT to_svector_query('simple', 'Dog & Cat);
SELECT to_svector_query('simple', 'Dog | Cat');
And for compute the distances, instead of the tsvector @@
operator, we colud use the existing sparse vector operators like neg inner prod <#>
SELECT text
FROM documents
ORDER BY sparse_keywords <#> to_svector_query('simple', 'Dog & Cat)
LIMIT 50;
Update
PostgreSQL ts_vector
already can handle a token/keyword weight. However this is a discrete weight of only 4 different options (A,B,C or D).
Lexemes that have positions can further be labeled with a weight, which can be A, B, C, or D. D is the default.
Weights are typically used to reflect document structure, for example by marking title words differently from body words.
source
I think it would be abstraction that sparse vectors become the missing ts_vector
with decimal (floating point) weigths, while keeping all the ts_query
capabilitites.
from pgvecto.rs.
Related Issues (20)
- feat: Implement RabitQ HOT 1
- docs: How to migrate from pgvector
- feat: implement vbase support for IVF
- feat: Python native SDK (without SQL) HOT 1
- bug: Potential memory leak issue in 0.3
- feat: fast scan for quantization
- feat: macos aarch64 support HOT 2
- feat: Disk format for RabitQ
- feat: Fast scan for RabitQ
- feat: Cosine and dot product distance for RabitQ
- Any plan to speed up pgvector.rs with Intel AMX instruction? HOT 4
- docs: Add a section about how to generate embedding in https://docs.pgvecto.rs/use-case/question-answering.html HOT 1
- Update sparse vector documentation
- Add pgvecto.rs to Trunk package registry HOT 3
- chore: Merge DiskANN from enterprise version
- pg_vectors index file name length breaks CNPG instance creation HOT 1
- indexing.hnsw.quantization.scalar does not improve the performance of query the similar embeddings
- bug: No such file or directory home/runner/work/pgvecto.rs/pgvecto.rs/crates/quantization/src/scalar/mod.rs HOT 3
- research: compatibility with citus HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pgvecto.rs.