Filtering outliers from Corpus with strange behavior about litstudy HOT 4 OPEN

Thanks for using litstudy!

Interesting problem, I'm not sure what is causing this problem. I'll look into this. The lack of proper tests for build_corpus and Corpus do not help, unfortunately :-(. Now might be to time to invest into those.

Look at the code, do you have any feeling on what problem could be? The only thing that look suspicious to me is the call to filter_extremes.

from litstudy.

Indeed. It seems that dic.filter_extremes(keep_n=max_tokens) is providing a similar functionality as preprocess_outliers(), so even if the preprocess_outliers() filter is behaving as expected (which I believe it is), once the filter_extremes() is called it overlaps the desired behavior.

I think it would be better to just keep filter_extremes() and incorporating the idea of using min_docs, max_docs and max_tokens in this method. I've checked the documentation and it might work:

Documenation: gensim.corpora.Dictionary.filter_extremes

from litstudy.

@stijnh, can you assign this issue to me? I'll look into it and try improve the tests for build_corpus

from litstudy.

Thanks for looking into this. I was not aware that filter_extremes would also filter tokens based on the number of documents

from litstudy.