serend1p1ty / core-pytorch-utils Goto Github PK

i have debug this problem for a long time, now i think the core reason is that saving too many state ,not only weights of model, but also optimizer state_dict.
if want to avoid oom during training, a solution is to decrease the batch size, but will cause a waster of gpu memory
is it better to save optimizer state stand alone?

In this way, we can directly install the cpu vai pip install xxx.

i want to use cpu to train a model on two machines with two gpus on each machine. but i have encounter a confused problem.
all processes hangs on dist.barrier() call in distributed.py file。

i have print some information to help judge. after print out "finish init_process_group, going to exec barrier", all 4 processes hangs.
then, i tried to remove dist.barrier()， it seems all right。could you give me some advice, how to debug this problem.

inner_iter starts from 0, I this this might be better:
process_string = f"Epoch: [{self.trainer.epoch}][{self.trainer.inner_iter + 1}/{self.trainer.epoch_len}]"