Resuming training about tinyllama HOT 8 OPEN

Hi, Artnoage. If you check the training log I actually resumed the process twice and did not notice any memory error. I am not sure why that is the case on your end.

Thanks for your suggestions on opening a discord. I think I will open one soon.

from tinyllama.

Yes I already knew that. I was just thinking maybe it had to do with the size of the network. like some missed parameter. When you did the first run, did you check the memory usage? If it is not a bit issue please leave the question open for a while, in case someone else tries it.

from tinyllama.

When you did the first run, did you check the memory usage?

The memory usage is always 39G on my end.

from tinyllama.

Hi! My training crashed, and I couldn't find the code to resume training from the last saved checkpoint. How can I resume my training? How do you handle this?

from tinyllama.

Hi! My training crashed, and I couldn't find the code to resume training from the last saved checkpoint. How can I resume my training? How do you handle this?

You can add --resume your_checkpoint.pth term in your pretraining command to resume training

from tinyllama.

Hi! My training crashed, and I couldn't find the code to resume training from the last saved checkpoint. How can I resume my training? How do you handle this?

You can add --resume your_checkpoint.pth term in your pretraining command to resume training

thanks, i found that

from tinyllama.

@artnoage can you please post this project I would like to try to he same with 2 4060 training at home.

from tinyllama.

from tinyllama.