Out of memory when training my own datasets about pytorch-cyclegan-and-pix2pix HOT 6 CLOSED

Could you share with us your training command? Could you train a pix2pix/cyclegan model on our provided dataset?

from pytorch-cyclegan-and-pix2pix.

@junyanz I have tried the following training command with your provided dataset:

facades dataset (400 training image pairs, 256×256):
python train.py --dataroot ./../datasets/facades --name facades_cyclegan --model cycle_gan --pool_size 50 --loadSize 128 --fineSize 128
Out of memory, when the loadSize and fineSize were further reduced to 64, out of memory disappeared

maps dataset (1096 training image pairs, 600×600):
python train.py --dataroot ./../datasets/maps --name maps_cyclegan --model cycle_gan --pool_size 50 --loadSize 128 --fineSize 128
Out of memory, when the loadSize and fineSize were further reduced to 64, out of memory disappeared

My own dataset (4821 training image pairs, 512×512):
python train.py --dataroot ./../datasets/modcn --name modcn_cyclegan --model cycle_gan --pool_size 50 --loadSize 128 --fineSize 128
Out of memory, when the loadSize and fineSize were further reduced to 64, out of memory disappeared

Besides, new errors appeared when loadSize and fineSize are set to 64. The error messages are repeated recurring during training without terminate the program...

Exception in user code: ------------------------------------------------------------
Traceback (most recent call last): File "/home/bwt/anaconda2/lib/python2.7/site-packages/visdom/__init__.py", line 228, in _send data=json.dumps(msg), File "/home/bwt/anaconda2/lib/python2.7/site-packages/requests/api.py", line 110, in post return request('post', url, data=data, json=json, **kwargs) File "/home/bwt/anaconda2/lib/python2.7/site-packages/requests/api.py", line 56, in request return session.request(method=method, url=url, **kwargs) File "/home/bwt/anaconda2/lib/python2.7/site-packages/requests/sessions.py", line 488, in request resp = self.send(prep, **send_kwargs) File "/home/bwt/anaconda2/lib/python2.7/site-packages/requests/sessions.py", line 609, in send r = adapter.send(request, **kwargs) File "/home/bwt/anaconda2/lib/python2.7/site-packages/requests/adapters.py", line 487, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /events (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f62f5560c50>: Failed to establish a new connection: [Errno 111] Connection refused',))

from pytorch-cyclegan-and-pix2pix.

When I try to use your Torch version CycleGAN, the program runs well during training

My own dataset (4821 training image pairs, 512×512):
DATA_ROOT=./../datasets/modcn name=modcn_model th train.lua loadSize=512 fineSize=512

I don't know why PyTorch and Torch have such great difference for CycleGAN.......

from pytorch-cyclegan-and-pix2pix.

1. GPU memory: Interesting. I ran the command on my machine, and only used 1294MB on my GTX 1080. Which pytorch version are you using now?

python train.py --dataroot ./../datasets/facades --name facades_cyclegan --model cycle_gan --pool_size 50 --loadSize 128 --fineSize 128

2. Error messages: Error message is related to visdom rather than GPU memory. You can start the visdom visualizatoin server by running python -m visdom.server. You can disable the visdom visualization by adding --display_id 0

from pytorch-cyclegan-and-pix2pix.

@junyanz Thanks for your help that the second problem was solved.
I'm using pytorch that depend on torch._version_=0.1.11+2b56711, the pytorch was installed a week ago so that it should be new. Now, I found when I killed all python process with command killall python, all the scripts aforementioned run successfully with loadSize=128 and fineSize=128, and the command nvidia-smi shows that 5033MiB/8110MiB for Memory-Usage.

But out of memory still raised with the default loadSize=286 and fineSize=256, scripts as follow:

python train.py --dataroot ./../datasets/facades --name facades_cyclegan --model cycle_gan --pool_size 50 --loadSize 286 --fineSize 256

from pytorch-cyclegan-and-pix2pix.

I am not sure why. I used 1300 MB GPU memory for 128/128, and 2218 MB GPU memory for 286/256.

from pytorch-cyclegan-and-pix2pix.