JY_about_regex_problem_in_bitextor_script about bitextor HOT 7 CLOSED

from bitextor.

Thanks for your reply.
Today I did an experiment on the sample dictionary (en-ca.dic)
I wrote a bash script to detect the following two regex expression, and then I found a problem described in the following():
tail -n +2 en-ca.dic | sed -r 's/^([^\s]+)\t([^\s]+)$/\2 @ \1/g' > output_s
tail -n +2 en-ca.dic | sed -r 's/^([^\S]+)\t([^\S]+)$/\2 @ \1/g' > output_S

the bash script described as follow (regex_test.sh):
TEST_DIC_NAME=en-ca.dic
tail -n +2 ${TEST_DIC_NAME} | sed -r 's/^([^\S]+)\t([^\S]+)$/\2 @ \1/g' > S_out_${TEST_DIC_NAME}
tail -n +2 ${TEST_DIC_NAME} | sed -r 's/^([^\s]+)\t([^\s]+)$/\2 @ \1/g' > s_out_${TEST_DIC_NAME}
echo "===================================="
echo "the total lines of the dictionary:"
wc -l ${TEST_DIC_NAME}
echo "===================================="
echo "the s_out_${TEST_DIC_NAME} result:"
echo "total lines containing the @ character:"
grep @ s_out_${TEST_DIC_NAME} | wc -l
echo "total lines not containing the @ character:"
grep -v @ s_out_${TEST_DIC_NAME} | wc -l
echo "===================================="
echo "the S_out_${TEST_DIC_NAME} result:"
echo "total lines containing the @ character:"
grep @ S_out_${TEST_DIC_NAME} | wc -l
echo "total lines not containing the @ character:"
grep -v @ S_out_${TEST_DIC_NAME} | wc -l

Do the above test, and I found that using the following may have better result:
"tail -n +2 en-ca.dic | sed -r 's/^([^\S]+)\t([^\S]+)$/\2 @ \1/g' > output_S "
But I don't konw the reason.

from bitextor.

from bitextor.

Thanks for your reply.
I see what you said, and I also confused about that.
And I do the expriment as you said : replace \s with [:space:] in the en-ca.dic , it really work properly.
But I do the expriment in my own dictionary: en-vi.dic , the result also very bad. I don't know why.
Following are my en-vi.dic file:

en-vi.txt

from bitextor.

from bitextor.

This is now integrated in masterand 6.x series branch. Release 6.0.1 incoming. If you find anything else wrong with this new expression, please, reopen the issue. Thanks for all your feedback!

from bitextor.

Thank you very much~

from bitextor.