Can't run on v3-8 or v3-32 TPU nodes. about compare_gan HOT 7 OPEN

You're welcome. And yes, you'll need to pip install googleapiclient and oauth2client as well.
For simplicity, you could also change
cp /usr/local/cuda-10.1/lib64/libcudart.so libcudart.so.10.0
(and other cp commands) to
sudo cp /usr/local/cuda-10.1/lib64/libcudart.so /usr/local/cuda-10.1/lib64/libcudart.so.10.0.

Otherwise, make sure you put the renamed libs on your LD_LIBRARY_PATH.

from compare_gan.

No problem, here's my basic setup:
VM Config

• c2-deeplearning-pytorch-1-4-cu101-v20200826-debian-10
• n1-highmem-8 (8 vCPUs, 52 GB memory)
• 1 NVIDIA Tesla V100
• Zone: us-central1-a
• Allow HTTP/HTTPS traffic

Obviously, the boot image is meant for PyTorch; you'll probably want to find something that comes with tensorflow-gpu==1.15 installed.

However, if go this route, make the following changes:

• On startup you'll be prompted to install nvidia drivers. Go ahead and install them (1 min)
• Run the following:
  sudo apt-get install python3 && sudo apt-get install pip3 && export pip=pip3 && export python=python3
pip install tensorflow-gpu==1.15.0
• Clone compare-gan, run pip install -e . as explained in the README

Because Tensorflow is...well, Tensorflow...your tensorflow-gpu 1.15 install will not actually use the gpu out of the box.
You'll need to copy or symlink a few libraries:

cp /usr/local/cuda-10.1/lib64/libcudart.so libcudart.so.10.0
cp /usr/local/cuda-10.1/lib64/libcublas.so.10 libcublas.so.10.0
cp /usr/local/cuda-10.1/lib64/libcufft.so.10 libcufft.so.10.0
cp /usr/local/cuda-10.1/lib64/libcurand.so.10 libcurand.so.10.0
cp /usr/local/cuda-10.1/lib64/libcusolver.so.10 libcusolver.so.10.0
cp /usr/local/cuda-10.1/lib64/libcusparse.so.10 libcusparse.so.10.0

TPU Config
Nothing hard here. I just used https://console.cloud.google.com/compute/tpus to create the latest set of TPU instances:

• zone=europe-west4-a (US zones work well too)
• TPU type=v3-32
• TPU software version: 1.15
• export TPU_NAME={your tpu name} and export TPU_ZONE={your zone}
• Make sure your model/data buckets have the right permission roles.

Anyway, you'll likely have more fun bugs to sort out for your particular use case. But this can at least get you started.

from compare_gan.

@mbbrodie Thanks for the information! Do you encounter the error message that request the installation of googleapiclient and oauth2client in your setup?

from compare_gan.

Same issue here. I tried TF 1.15 and TF 1.15.3 and all got the same error message.

from compare_gan.

@hytseng0509, found a workaround for now:
After running into errors on different VM configs, I got TPU training (no eval) working on a VM node with a GPU. Regardless of the TF/Python version or RAM/# CPUs, I couldn't get training to work without a GPU on board.

For the repo maintainers:
(First, thank you -- the repo is awesome and much more intuitive to use than TF-GAN).

Some of the README could use some updated language:

• https://github.com/google/compare_gan#training-on-cloud-tpus seems to imply that you ONLY need a GPU if you're evaluating.
• Listing the TF & Python versions (that you used to test/run your models) would be helpful.
• A summary for setting up on GCP would be nice (e.g. getting started with a basic VM and v3-8 TPU).

I imagine newcomers (especially students!) would appreciate spending their TPU $$$'s actually training instead of installing 10 different TF/TPU setups.

from compare_gan.

@mbbrodie Thanks for sharing! How do you config the VM and TPU? I still got the same error message using a VM with GPUs.

from compare_gan.

I have the same error here related to InvalidArgumentError: Cannot assign a device for operation input_pipeline_task0/TensorSliceDataset. Wonder if this is related to the TF framework?

from compare_gan.