Make a mode that translates line by line about apertium HOT 8 CLOSED

I think the best I've been able to make is to work with some sed hack like sed -e 's/$/§§/' | apertium -f line | sed -e 's/§.*//'

from apertium.

We need proper block separation for markup handling anyway, similar to null-flush mode. Proper line handling could be done with that work.

from apertium.

@flammie even that doesn't work sometimes ...

from apertium.

This remains unchanged even with the word-bound blank work, because word-bound blanks are an orthogonal problem. WBBs must move/split/merge with the token, while what we need here is something that acts like a hard sentence boundary but not a hard context boundary. And normal whitespace should still behave as it currently does - staying sort of in place.

In essence, we need what in CG terms would be delimiters. Something that rules cannot move words across without being explicitly allowed to do so, but where one can still inspect the other sentences for anaphora and context.

For markup handling, we reused null-flush because it is the only thing that is currently safe enough to strictly keep blocks from interacting. Non-null delimiters would need another code point, and it would probably be a significant change to how rules are run.

from apertium.

After the latest update to blanks, it is guaranteed to have the same number of line breaks in the transfer input as the output, which is where the line breaks usually used to disappear (or add).

from apertium.

I think recursive does this now as well (@mr-martian ?), so we can test this and close it.

from apertium.

recursive blank handling has been updated and should be approximately equivalent to t*x blank handling (it frequently deals with much longer sequences than t1x, so the blanks wind up offset by more, but the basic algorithm is the same)

from apertium.

Somehow or other I'm still getting a different number of line breaks in tx for br-fr (and I can be sure it's tx because rtx in an otherwise identical pipeline doesn't do that).

I think at present -anaphora is the only module that would ever need to look at multiple lines, so what if -wblank-mode had an option that would give a different argument to -anaphora so it could look across nulls?

On the other hand, what I have now several times found myself wanting is a way to translate each line of a file independently, so maybe -anaphora shouldn't be treated differently (or maybe we need 2 different modes).

from apertium.