CUDA out of memory. Tried to allocate 18.19 GiB (GPU 0; 10.76 GiB total capacity; 2.85 GiB already allocated; 1.96 GiB free; 296.54 MiB cached) about se-ssd HOT 7 CLOSED

That's the same problem as mine.

At early steps, there are many positive boxes, which results in the super large cuda memory problem.

from se-ssd.

Please use a pre-trained model to initialize both teacher and student SSD.

from se-ssd.

That's the same problem as mine.

At early steps, there are many positive boxes, which results in the super large cuda memory problem.

Has the problem been solved?

from se-ssd.

done

from se-ssd.

Traceback (most recent call last):
File "train.py", line 124, in
main()
File "train.py", line 119, in main
train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate,logger=logger,)
File "/media/ubuntu-502/pan1/tony_data/SESSD/SE-SSD-code/det3d/torchie/apis/train_sessd.py", line 327, in train_detector
trainer.run(data_loaders, cfg.workflow, cfg.total_epochs, local_rank=cfg.local_rank)
File "/media/ubuntu-502/pan1/tony_data/SESSD/SE-SSD-code/det3d/torchie/trainer/trainer_sessd.py", line 472, in run
epoch_runner(data_loaders[0], data_loaders[1], self.epoch, **kwargs)
File "/media/ubuntu-502/pan1/tony_data/SESSD/SE-SSD-code/det3d/torchie/trainer/trainer_sessd.py", line 355, in train
self.call_hook("after_train_iter") # optim_hook: backprop;
File "/media/ubuntu-502/pan1/tony_data/SESSD/SE-SSD-code/det3d/torchie/trainer/trainer_sessd.py", line 210, in call_hook
getattr(hook, fn_name)(self) # self is the param (trainer/runner) of func hook.fn_name
File "/media/ubuntu-502/pan1/tony_data/SESSD/SE-SSD-code/det3d/core/utils/dist_utils.py", line 53, in after_train_iter
runner.outputs["loss"].backward()
File "/media/ubuntu-502/pan1/tony_data/SESSD/sessd/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/media/ubuntu-502/pan1/tony_data/SESSD/sessd/lib/python3.6/site-packages/torch/autograd/init.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

I had downloaded this pre-trained model, but it appears above problem. I set this batch_size=1, but it can't be solved. Who have a good idea.

from se-ssd.

hi @zyxcambridge , how did you solve the problem? I set "load from = se-ssd-model.pth", but it still crash at 50 epoch

from se-ssd.

I encountered the same problem. I added the following code to trained_sessd.py in line 297 to solve this problem

"torch.backends.cudnn.enabled = False"

rusult show in:
if distributed:
model = apex.parallel.convert_syncbn_model(model)
model = DistributedDataParallel(
model.cuda(cfg.local_rank),
device_ids=[cfg.local_rank],
output_device=cfg.local_rank,
# broadcast_buffers=False,
find_unused_parameters=True,
)
else:
model = model.cuda()

torch.backends.cudnn.enabled = False

logger.info(f"model structure: {model}")

model_ema = copy.deepcopy(model)

from se-ssd.