###################################################################### Description of the program:

Seperates all punctuations from the words. Also daMDa and double daMDa in unicode Indic scripts (especially Hindi).

Encoding supported: UTF-8 only.

###################################################################### How to run

$perl tokenizer.pl --input foo --output bar

input is a required field.

output is optional - default extension: ".tok" to input file,

if --output option is missing

######################################################################

Thank You. Suggestions are Welcome: [email protected]