The fmdl from wang-h

fmdl's People

Stargazers

Watchers

fmdl's Issues

Segmentation only by letters

Hello there,

I've tested the code to segment protein aminoacid sequences (it's similar to chinese language rules).

When I run the code, it gave letter by letter segmentation of the sequences,

Is the code still under development, or is there any limitations regarding input sequences,

Appreciate any help, thanks !

"ValueError: math domain error" when training on a large file

I have a large training file with 1252976 lines. When I use it to train a model(vocab_size: 30000), it raised an error like following:

Building pair statistics: 2116233it [00:03, 614115.37it/s]
Commit to codebook:   8% 688/8761 [00:00<00:10, 753.77it/s]
Traceback (most recent call last):
  File "learn_mdl.py", line 202, in <module>
    main(args)
  File "learn_mdl.py", line 161, in main
    codebook = trainer(args.iterations, args.verbose)
  File "learn_mdl.py", line 150, in train
    if not self.update_codebook(candidates):
  File "learn_mdl.py", line 129, in update_codebook
    if self.commit_and_success(pair, total, cost):
  File "learn_mdl.py", line 106, in commit_and_success
    if sum(self.compute_cost(w1, w2, total)) > 0:
  File "learn_mdl.py", line 63, in compute_cost
    float(total), float(c1), float(c2), self.data_len)
  File "learn_mdl.py", line 68, in compute_data_cost
    data_cost += c1 * math.log(c1 / n)
ValueError: math domain error

How can I fix it? Thanks in advance

Recommend Projects