Sparse vectors: 1 representation, 2 use cases about pgvecto.rs HOT 2 OPEN

Thanks for your suggestion! We're aware of the sparse index here and have some prototype at #552. For the weighted keyword search, we have tried https://github.com/tensorchord/pg_bestmatch.rs. Does it work for you? We're also thinking of a better way to directly build text index for bm25 search

from pgvecto.rs.

Thanks for aswering, i will take a look at pg_bestmatch.rs.

I think that part of storing sparse vectors are solved. However i think, the query part must be rethinked.

I honestly think that the native to_tsquery API (link1 link2) is great and can be replicated for sparse vectors (maybe to_svector_query). The to_tsquery API offers great flexibiliy (boolean operators):

• SELECT to_tsquery('english', 'The & Fat & Rats');
• SELECT to_tsquery('english', 'Fat | Rats');
• SELECT to_tsquery('simple', 'Fat | Rats');

I think this type of API is great and can solve the previous animal keywors example, and also instead of 'english' we can add custom sparse encoders (TFIDF, BM25, SPLADE, CUSTOM)

• SELECT to_svector_query('tfidf', 'We begin, as always, with the text.');
• SELECT to_svector_query('bm25', 'We begin, as always, with the text.');
• SELECT to_svector_query('simple', 'Dog & Cat);
• SELECT to_svector_query('simple', 'Dog | Cat');

And for compute the distances, instead of the tsvector @@ operator, we colud use the existing sparse vector operators like neg inner prod <#>

SELECT text
FROM documents
ORDER BY sparse_keywords <#> to_svector_query('simple', 'Dog & Cat)
LIMIT 50;

Update

PostgreSQL ts_vector already can handle a token/keyword weight. However this is a discrete weight of only 4 different options (A,B,C or D).

Lexemes that have positions can further be labeled with a weight, which can be A, B, C, or D. D is the default.
Weights are typically used to reflect document structure, for example by marking title words differently from body words.
source

I think it would be abstraction that sparse vectors become the missing ts_vector with decimal (floating point) weigths, while keeping all the ts_query capabilitites.

from pgvecto.rs.