Why need the data processing step? about adversarial_video_generation HOT 3 CLOSED

Here is my hypothesis:
Once the dataset is expressive enough for the problem, it is more efficient to train the network over that dataset multiple times (i.e. epochs). In this case, preprocessing the data multiple times is unnecessary.

from adversarial_video_generation.

Sorry for the delayed response! The initial reason I added a separate pre-processing pipeline was because I was being bottlenecked by i/o and it is much more efficient to read in 32x32 image patches than the full-sized images. I don't remember what the memory specs of my machine were, but I think I was running into issues loading the whole dataset into memory. If you are able to do that, it might make more sense to create the patches at runtime.

edit
One problem with creating the patches at runtime on the Ms. Pac-Man data is that most patches will have no movement in them. To solve this, I randomly sample patches until it finds one with movement. This means that it will take longer to generate some patches than others. It's probably more efficient to get all of this generation cost out of the way once during pre-processing than having to deal with it every epoch during training.

from adversarial_video_generation.

Thanks for your reply? It's very helpful.

from adversarial_video_generation.