LiteralNER shall tokenize each entry about iepy HOT 1 OPEN

For documentation's sake,, something we talked in real life:

Perhaps tokenization, pos-tagging, ner-tagging and segmentation could be non-optional parts of the preprocessing pipeline since iepy's core would break without them anyway.

If those parts are non-optional, then they can be passed as kwargs to the preprocessing pipeline and therefore it can be easily ensured that tokenization is the same for both documents and literal tagging.

from iepy.