Don't force sim to score bogus terms (e.g. docfreq=0) [LUCENE-8020] about stargazers-migration-test HOT 6 CLOSED

I noticed this when trying to debug AfterEffectB in LUCENE-8015.

The formula should be: (F + 1) / (n * (tfn + 1)) but we currently use (F + 1 + 1) / ((n + 1) * (tfn + 1)) and I couldn't remember why we had this mess everywhere.

[Legacy Jira: Robert Muir (@rmuir) on Oct 28 2017]

from stargazers-migration-test.

Here is a patch. core tests pass. I didn't yet revert all the similarity formula hacks as that will take more time.

[Legacy Jira: Robert Muir (@rmuir) on Oct 28 2017]

from stargazers-migration-test.

Updated patch with javadocs improvements for Term/CollectionStatistics. solr distributed idf code needed small tweaks but they were easy because it already assumed this method might return null.

[Legacy Jira: Robert Muir (@rmuir) on Oct 28 2017]

from stargazers-migration-test.

I reviewed callers of termStatistics and found also TermAutomatonQuery in sandbox (scores like phrase query but has the same current issue as SpanOrQuery if some don't exist), fixed it the same way, and added unit tests. I think its ready.

[Legacy Jira: Robert Muir (@rmuir) on Oct 28 2017]

from stargazers-migration-test.

Commit e0bde579815ae5ce2525bb659d04e908812f1605 in lucene-solr's branch refs/heads/master from @rmuir
https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e0bde57

LUCENE-8020: don't force sim to score bogus terms (e.g. docfreq=0)

[Legacy Jira: ASF subversion and git services on Oct 31 2017]

from stargazers-migration-test.

I spun off LUCENE-8023 and LUCENE-8024 to simplify sim impls.

[Legacy Jira: Robert Muir (@rmuir) on Oct 31 2017]

from stargazers-migration-test.