This would require a longlat field type... (and coords if we want simple 2D as well

Then the user would want to perform queries such as

• return document at a distance of less than X km
• return document within this polygon

Sorting documents by a given field can open the door to various optimization

• sort by score related metric
• group by/collapse by a sorted field
• locality
• range queries

Publishing and committing don't have to be the same.

In many usage, users want document to be searchable as soon as possible.
Let's call making a segment ready for search "publishing.

Committing on the other hand is about order consistency, (after commit, a document is
searchable iff it was added before the commit) and more importantly persistence.

The current json parser does not allow for multivalues.

{
 "author": ["Les Paul", "Mary Ford"]
}

should be parsed correctly

Schema is copied quite often.

Maybe add an property to Index, and wrap it in an Arc could be a solution

Implement a merge policy inspired by LogMergePolicy from Lucene.

Segment size will be determined by the number of documents.

Unless I am mistaken, this merge policy should not need to store any state.

Currently we do not handle removing the segments.
This is a bit tricky because the file (or rather ReadOnlySource) involved in a search should outlive the searcher.

Currently we decompress / recompress docs when we merge segments.

+barack obama should translate in (title:barack OR body:barack) AND (title:obama OR body:obama)

Currently, each indexing thread writes to disk when its indexing buffer is full. As they start in sync, this means they tend to decide to spill their segment all at the same time.
This is suboptimal, as the disk is idle for a while, and then is asked to concurrently write a huge amount of data.

IndexReader from different processes should see the changes to the meta.json file.

Until now tantivy only handled adding document. This issue is about adding delete. Provided a unique id term exists in the schema, update can then be done on the application side by expanding it into a delete and an add.

Since adding a document will not be the only operation anymore, "docstamp" needs to be renamed "opstamp"
For each segment, deletes are represented as a tombstone .del file that stores the bitset of documents that are deleted.

For this first iteration, only terms can be deleted. When a term is added for deletion, it is
appended to the delete queue without being processed.

Committed segments

Upon commit, all previously committed segment have deletes up to date up to the last commit opstamp.
Their .del file therefore needs to be updated up to the new commit opstamp.

Uncommitted segments

These uncommitted segments come with a transient information which is the opstamp up to which deletes have been processed, which is not necessarily the last commit opstamp.

(All of this info being transient is ok as uncommitted segments are by nature transient. If the indexing process crashes and restart, indexing must resume from the last commit opstamp, and there is no way to use the uncommitted segments.)

When a segment being written gets closed, the deletes up to the last opstamp are read to build a bitset of the deleted documents, create a mapping of the compacted doc ids, and the finalized and flushed segment already takes in account the delete.

Because deletes should only apply to documents that where added before the delete, a transient mapping document -> docstamp. The function is monotonic so identifying the document that should be affected by a delete is a matter of binary searching the delete opstamp in this mapping, identifying the smallest doc id with a greater docstamp, reading the inverted list from the segment writer and marking as deleted all the doc ids greater or equal to this limit docstamp.

The delete opstamp at this point is the last opstamp of the segment.

Merging segments

When merging uncommitted segments, all of the segments tombstone bitset are updated to a common opstamp. (When merging committed segments, they are already at the same docstamp so this is not required).
Similarly to what is done with writing segments, we create a map of old docid -> new docid and phyiscally remove deleted docids.
After the merged segment is flushed, we check if its delete opstamp is up to date with the committed delete opstamp. If not, we compute and create the tombstone file accordingly.

Dumping delete terms from Memory

The delete queue is a linear queue in which we can easily iterate over sections defined using timestamps.
The timing at which the queue can be purge is non trivial as the last consumer of it can be the commit or a currently running merge.

Client of the delete queue can suscribe to the queue and express that they will need to consume
it from a given starting point in the future.

For the first version, there will be no mechanism to purge the delete queue before commit, when the queue is getting too big.

... by a mmapped skip list.

Refactor Query to introduce Weight or Scorer like Lucene.

... and move the gigantic example in rustdoc to this instead.

This will most likely require a variable size DocValue.

There should be only one IndexWriter working at a time (within our process, or within another process). We should probably add a lockfile system to prevent people from making a mistake.

The current positions encoding is very slow, but we should already be able to implement PhraseQuery on top of it.

Currently we don't do anything smart when we don't take advantage of skipping when we have a must terms.

It would be great to see track periodically our performance and ideally compare it to Lucene.

Maybe using the same dataset as
http://home.apache.org/~mikemccand/lucenebench/

Currently users are expected to manually merge segments. It would be nice to have some kind of merge strategy working out of the box

It might be possible if we pass the heap around at each call.

Reason unknown ... probably the skip list merge.

Happens on the index wikipedia example

Add a asciicinema terminal screencast to the tutorial

Tantivy automatically commits new segment. This prevents anything serious from being build on top of tantivy.

flame keeps thread local "tracks" and has a method to clear context, so there is no need to pass anything as an argument.

Externalize the searchers to a IndexReader object, and watch for changes in meta.json

The role of the QueryParser is to give an off-the-shelf solution to people wanting to deliver a search box to end user.
Currently QueryParser::parse can return an error if the query does not match our query grammar.

e.g.: +(happy : the parenthesis is not closed.

This is great, but how should somebody writing a site handle this error?
Maybe for a log analysis search engine, displaying an explicit syntax error is a good idea, but for most use case, (ecommerce, document search, etc.) we want to never return an error and just do our best to handle the user query.

For this reason, we want to add a new parse_lenient that would never fail and always return a result.

pub fn parse_query_lenient(&self, query: &str) -> Box<Query> {
    // ...
}

The behavior of this lenient mode does NOT have to be very smart...
For instance, it could be a second very naive parser taking over when the initial parser failed.

For instance, this naive parser could remove all special characters and run the initial parser.

Smarter attempts to interpret the faulty user query are also welcome, as long as we return Box<Query> and we are confident the code will never panic.

If someone can define collectors as:

• docset -> A
• [A] -> B or ( A,A -> B and B::zero)

It would make distribution or concurrence trivial.

SIMD is not portable as is on ARM. Also this would open the door to compile tantivy to web assembly.

Do you think it is possible to use tantivy to measure the similarity between texts?

First byte as term can be very limitting. It will prevent building dynamic schemas (we might want to define a schema with wildcard fields : text_*: {indexing_options: ...}

fst now allows to work withArc<Vec<u8>>

BurntSushi/fst#11

Removes the crazy "let's copy everything all the time" in the ram directory

High frequency words like "the" are hurting the query performance.

They are most of the time not queried alone.
Build the analyzer that generate the right tokens for index, and query.

In query

 "the lord of the ring" -> "the lord", "of", "the", "ring"

In index

 "the lord of the ring" -> [ ["the", "lord"], "the lord"], "of", "the", "ring"

They are here to make the segment lifetime last the running queries.
The API should allow the index to advertise for new segments rapidly.

running: "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-g" "-m64" "-fPIC" "-I" "./cpp/simdcomp/include" "-std=c++11" "-O3" "-mssse3" "-o" "/home/maciej/git/tantivy/target/debug/build/tantivy-c843e95d6308187a/out/cpp/simdcomp_wrapper.o" "-c" "cpp/simdcomp_wrapper.cpp"
cargo:warning=cpp/simdcomp_wrapper.cpp:4:22: fatal error: simdcomp.h: No such file or directory
cargo:warning=compilation terminated.
ExitStatus(ExitStatus(256))

How should I install it? Documentation doesn't mention it.

Vec is probably not the best datastructure to store the postings for the segment writer.
What we probably want is a linked list of blocks.

Extra points if the blocks are compressed themselves every time they are "closed".

Can I use tantivy to index french text, eg. with the articles of French wikipedia?

This would be handy for unit tests for instance.

Maybe something like :

doc!( field1=>"toto", field2=>5, )

Value implements From<String> and From<u32> so that should be feasible.

The current query explain uses term1, term2, etc. instead of the actual terms.

Using terms would be nicer.