Cannot combine Text models if they contain ()"'[] in the string about markovify HOT 2 CLOSED

Hi @Denton-L! Thanks for raising this issue.

It looks like the error actually comes when the model is created rather than at the combine step. (See the first three three lines of the stacktrace you've pasted; this can also be confirmed by removing the combined_model = ... line from your script and re-running it.)

Here's what seems to be happening:

• When new_model = markovify.NewlineText(broken) is invoked in your test script, it is trying to create a Markov model with just one line of text (e.g., 'this string is not broken'). When using NewlineText, one line of text corresponds to a single sentence.

• When markovify.Text — of which markovify.NewlineText is a subclass — ingests a new corpus, it ignores sentences with certain characters/patterns. (See reject_pat here.)

• If a corpus is composed entirely of sentences containing those patterns — as the error-triggering examples above do — then it effectively is working with an empty corpus, which is causing this error. You can test this by, for example, changing "this string contains (" to "this string contains ( \n this is an example sentence".

If this is causing problems with one of your projects, the easiest fix would be to override markovify. Text.test_sentence_input(...).

In the slightly longer-term, I think these fixes are in order:

• A more informative error message when all sentences of a corpus are rejected.

• An easier-to-override reject_pat in markovify. Text.test_sentence_input(...)

Any other thoughts/suggestions?

from markovify.

@jsvine

A more informative error message when all sentences of a corpus are rejected.

👍

It would be great if we can add other punctuation marks. (e.g. Japanese: 。)

from markovify.