[BUG] IDF change to 0 after writing the same text three times about redisearch HOT 6 OPEN

Thanks @meiravgri . Any update on addressing this issue is highly appreciated.

from redisearch.

@freecw Hi.
Indeed looks like a bug! Good catch!
Thanks.

from redisearch.

@meiravgri it seems the "IDF" will change to 0 after we deleting another key

from redisearch.

I've found the root cause for this bug and opened an internal ticket to our team.
We'll address this asap. Thanks for reporting.

from redisearch.

@meiravgri Thanks, could you please share the root cause? And any expected time for addressing this?

from redisearch.

sure
IDF score depends on sp->docs.size and idx->numDocs like so:

logb(1.0F + sp->docs.size / (idx->numDocs ? idx->numDocs : (double)1));

Since idx->numDocs reflects the number of valid docs + updated docs that weren’t yet cleaned by the gc, AND the default gc threshold is 100, the score is calculated based on wrong index statistics.

As a temporary workaround you can decrease the the gc threshold:
FT.CONFIG set FORK_GC_CLEAN_THRESHOLD 0

from redisearch.