GPU Implementation runs no faster than CPU counterparts about cleanrl HOT 3 CLOSED

On environment that do not require CNN, that is to be expected.

The GPU advantage seems to only kick in when the data / network size crosses a certain threshold,
namely where the overhead of moving the data to the GPU is negligible when compared to the cost of doing the feed forward as well as back propagation operations.

The following Wandb report only investigates the batch size, but the cuda=True version seems to become overcome the CPU version around batch size ~7500:
https://app.wandb.ai/dosssman/cleanrl-cpu-gpu-comp/reports/Time-taken-w.r.t.-device--VmlldzoxMzIwNzE

Just to be sure, still need to check without the Wandb logging to make sure it is consistent without the logging overhead.

from cleanrl.

Well, turns out the Wandb overhead seems to really mess up the results.
When running the experiments with source code on a tmpfs device, the GPU version catches up even faster to the CPU one.

Although incomplete at the time of this writing, the following Wandb Report shows that GPU starts showing faster runtime than CPU from ~512 batch size:
https://app.wandb.ai/dosssman/cleanrl-cpu-gpu-comp-2/reports/Untitled-Report--VmlldzoxMzI0NjU

from cleanrl.

This has largely been fixed with the latest commits. Closing the issue for now.

from cleanrl.