Spacy Integration - "detect an empty sentence" about spanmarkerner HOT 4 OPEN

Actually, using the en_core_web_lg model instead of the sm model mitigates the problem to a great extent (it just parses the sentences better). May not be worth a pull just yet, but I had to fix the same issue during training by pre-processing the sentences.

from spanmarkerner.

Hello!

I don't think there's a workaround yet. Feel free to make a pull request and I'll try to have a look at it :)

• Tom Aarsen

from spanmarkerner.

Hi,

I have a one-line fix that should work for this as it keeps happening. spacy integration code that resolves this issue fairly nicely. I am not sure how to do the pull request, but the in the "spacy_integration.py" file I would propose the following change:

    def __call__(self, doc: Doc) -> Doc:
        """Fill `doc.ents` and `span.label_` using the chosen SpanMarker model."""
        sents = list(doc.sents)
        inputs = [[token.text if not token.is_space else "" for token in sent] for sent in sents]
        # Remove any sentences where the tokens are all empty strings or the sentence has 0 tokens.
        inputs = [sentence for sentence in inputs if any(sentence) and len(sentence) > 0]

        # use document-level context in the inference if the model was also trained that way
        if self.model.config.trained_with_document_context:
            inputs = self.convert_inputs_to_dataset(inputs)

I have updated the code and run a substantial amount of text through it. I did not however create any unit-tests but will be happy to do so you would consider this addition. It could be merged into the list comprehension above, but I wanted to show the logic clearly here.

from spanmarkerner.

The pipe() function seems a bit difficult to fix, because if empty sentences are removed it will be tricky to calculate the offset value. Couldn't the tokenizer code somehow ignore empty sentences?

from spanmarkerner.