Jamspell not able to train model for corpus larger than a certain size about jamspell HOT 6 CLOSED

Just as a small benchmark: It looks like for a 15GB txt file as input, you need about 300GB RAM while loading and about 50 GB RAM while processing. I don't know if these values are influenced by the OS or anything else.

from jamspell.

Same issue, but on just 1.8 GB. Any workarounds?

from jamspell.

Такая же ошибка на корпусе в 310 MB, есть какое-нибудь решение?

UPD1. Всё оказалось банально и просто - не хватило свободной оперативной памяти. Для обучения на текстовом файле размером 310 MB понадобилось 5.6 GB оперативной памяти.

UPD2. Однако для оценки полученной модели потребовалось 15.1 GB оперативной памяти. Зато значение ошибки получилось 1.8% :)

from jamspell.

можно свап попробовать врубить

from jamspell.

@Niecke thanks for this benchmarks. So basically it's pretty hard to train a new model outside a scalable infrastructure like AWS EC2 instances, etc. I mean I do not have a 300GB RAM Xeon under my desk :D

from jamspell.

1. Try to increase swap
2. Try a new "JamSpellPro" version. It has optimization for training large model (it use only 1/2 of original memory usage).

JamSpellPro is available at jamspell.com

from jamspell.