Comments (7)
You're welcome. And yes, you'll need to pip install googleapiclient
and oauth2client
as well.
For simplicity, you could also change
cp /usr/local/cuda-10.1/lib64/libcudart.so libcudart.so.10.0
(and other cp commands) to
sudo cp /usr/local/cuda-10.1/lib64/libcudart.so /usr/local/cuda-10.1/lib64/libcudart.so.10.0
.
Otherwise, make sure you put the renamed libs on your LD_LIBRARY_PATH.
from compare_gan.
No problem, here's my basic setup:
VM Config
- c2-deeplearning-pytorch-1-4-cu101-v20200826-debian-10
- n1-highmem-8 (8 vCPUs, 52 GB memory)
- 1 NVIDIA Tesla V100
- Zone: us-central1-a
- Allow HTTP/HTTPS traffic
Obviously, the boot image is meant for PyTorch; you'll probably want to find something that comes with tensorflow-gpu==1.15 installed.
However, if go this route, make the following changes:
- On startup you'll be prompted to install nvidia drivers. Go ahead and install them (1 min)
- Run the following:
sudo apt-get install python3 && sudo apt-get install pip3 && export pip=pip3 && export python=python3
pip install tensorflow-gpu==1.15.0
- Clone compare-gan, run pip install -e . as explained in the README
Because Tensorflow is...well, Tensorflow...your tensorflow-gpu 1.15 install will not actually use the gpu out of the box.
You'll need to copy or symlink a few libraries:
cp /usr/local/cuda-10.1/lib64/libcudart.so libcudart.so.10.0
cp /usr/local/cuda-10.1/lib64/libcublas.so.10 libcublas.so.10.0
cp /usr/local/cuda-10.1/lib64/libcufft.so.10 libcufft.so.10.0
cp /usr/local/cuda-10.1/lib64/libcurand.so.10 libcurand.so.10.0
cp /usr/local/cuda-10.1/lib64/libcusolver.so.10 libcusolver.so.10.0
cp /usr/local/cuda-10.1/lib64/libcusparse.so.10 libcusparse.so.10.0
TPU Config
Nothing hard here. I just used https://console.cloud.google.com/compute/tpus to create the latest set of TPU instances:
- zone=europe-west4-a (US zones work well too)
- TPU type=v3-32
- TPU software version: 1.15
- export
TPU_NAME={your tpu name}
andexport TPU_ZONE={your zone}
- Make sure your model/data buckets have the right permission roles.
Anyway, you'll likely have more fun bugs to sort out for your particular use case. But this can at least get you started.
from compare_gan.
@mbbrodie Thanks for the information! Do you encounter the error message that request the installation of googleapiclient
and oauth2client
in your setup?
from compare_gan.
Same issue here. I tried TF 1.15 and TF 1.15.3 and all got the same error message.
from compare_gan.
@hytseng0509, found a workaround for now:
After running into errors on different VM configs, I got TPU training (no eval) working on a VM node with a GPU. Regardless of the TF/Python version or RAM/# CPUs, I couldn't get training to work without a GPU on board.
For the repo maintainers:
(First, thank you -- the repo is awesome and much more intuitive to use than TF-GAN).
Some of the README could use some updated language:
- https://github.com/google/compare_gan#training-on-cloud-tpus seems to imply that you ONLY need a GPU if you're evaluating.
- Listing the TF & Python versions (that you used to test/run your models) would be helpful.
- A summary for setting up on GCP would be nice (e.g. getting started with a basic VM and v3-8 TPU).
I imagine newcomers (especially students!) would appreciate spending their TPU $$$'s actually training instead of installing 10 different TF/TPU setups.
from compare_gan.
@mbbrodie Thanks for sharing! How do you config the VM and TPU? I still got the same error message using a VM with GPUs.
from compare_gan.
I have the same error here related to InvalidArgumentError: Cannot assign a device for operation input_pipeline_task0/TensorSliceDataset
. Wonder if this is related to the TF framework?
from compare_gan.
Related Issues (20)
- resnet_biggan_deep seems not consist with BigGanDeep in the paper HOT 1
- Is the widening factor of S2GAN's feature extractor really 16? HOT 4
- SSGAN and unsupervised S3GAN are unavailable on tfhub.dev
- Need trainable version for pretrained SSGAN in tfhub HOT 2
- train on GPU HOT 1
- The SSGAN model has no rotation loss in discriminator HOT 1
- Problem about shuffle buffer size for unconditional GAN HOT 1
- issue: ModuleNotFoundError: No module named 'compare_gan' HOT 1
- http error 409 conflict HOT 1
- Error "Retval[7] does not have value" when training SSGAN HOT 3
- celeba_hq dataset was removed?
- SNDCGAN implementation is different from the architecture described in SN paper HOT 1
- Detailed procedure for preprocessing CelebA-HQ 128x128 images HOT 2
- module 'tensorflow' has no attribute 'GraphKeys' HOT 1
- Can I use a custom dataset HOT 1
- TypeError: 'NoneType' object is not iterable HOT 2
- Details in the implementation of BigGAN HOT 2
- tf-nightly-gpu version < 2 no longer exists
- Critical error in the implementation of compare_gan vs the official BigGAN model
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from compare_gan.