Document aligner happily returns nothing with piped input about bitextor HOT 1 CLOSED

It indeed uses two passes to compute the idf score in the first pass. I experimented with storing all documents in memory during this read, but that didn't lead anywhere good. I suppose having the gzipped document in the file cache is more effective.

I can think of two ways of solving this:

1. https://stackoverflow.com/questions/21468856/check-if-file-is-a-named-pipe-fifo-in-c/21468960#21468960
2. compare the number of lines read in the first and second pass

Option 1 can raise an error on start-up, no need to read once and only then error. Option 2 can also detect a file that's still being written or changed due to other reasons between the first and second pass.

Or both.

Any preferences?

from bitextor.