I'm running a sentiment type training environment with about 400,000 samples. The text is short, at most 255 characters each, but most are much less.
I'm running 32G on Linux running without a GUI and few other processes running. PHP is set to memory_limit -1.
Linux routinely kills my PHP process due to lack of memory. Following this guide I've set my overcommit_memory value to 2 which allows the script to run a little longer but it eventually runs out of memory and stops.
I've ordered 32G of more RAM hoping this will help with my problem, but, are there any other techniques I could use to reduce memory usage? Or, is there a way to see/calculate how much memory is needed before even attempting to run the script?