Comments (3)
dav-ell
commented on May 9, 2024
Can we add support for Google Cloud TPUs as well?
from deepspeed.
jeffra
commented on May 9, 2024
@dav-ell as far as I understand, TPU support with PyTorch requires XLA (see: https://github.com/pytorch/xla). We haven't tested DeepSpeed with XLA. Also we don't have access to any TPU's to test this on but if you (or anyone else does) we welcome pull requests or feedback about issues along this direction.
from deepspeed.
dav-ell
commented on May 9, 2024
@jeffra Appreciate this. I have access to TPUs for about a month, but don't anticipate being able to get to any PRs. I'll probably just be sticking with XLA as long as I can -- but the easy scaling from DeepSpeed will be missed.
from deepspeed.
Related Issues (20)
- [REQUEST] detect opbuilder list at launch time HOT 7
- [BUG] FP32 gradient accumulation result in crash. HOT 2
- [BUG]Training speed of deepspeed>=0.12.5 becomed slower than before! HOT 9
- The sequence length is not divisible by Sequence Parallel World Size
- [REQUEST] parameter sharding, gradient sharding, and optimizer state sharding with various sharding factors like Zero++
- RuntimeError: You can't move a model that has some modules offloaded to cpu or disk. HOT 1
- [BUG] `deepspeed.zero.Init` leaks
- AFAIK if I run the `install.sh` script with `curl` before installing dos2unix, it will fail, so dos2unix should be installed before the `curl`. HOT 2
- [BUG] Differences between training result using zero-2 and zero-3
- [BUG]Cannot install deepspeed when cuda is installed by a non-root user
- Compilation Errors with DeepSpeed on Multi-GPU Training Setup HOT 1
- [BUG] 13.3 version pre-build Compilation error `fused_adam_frontend.o: No such file or directory` and `multi_tensor_adam.o: No such file or directory` HOT 3
- [BUG] ValueError: `.to` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
- [BUG] Memory leak during autograd backwards
- [BUG] access the gradient while tranining
- [BUG] Errors when running with Microsoft Phi models HOT 2
- [BUG] reduce scatter cannot be overlap when using zero HOT 2
- xpus not detected[BUG] HOT 5
- [BUG] Deepspeed Crashes when using MoE, Stage 2 Offload with DeepSpeedCPUAdam HOT 3
- [BUG] Deepspeed repeatedly requests to lock files and gets stuck
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepspeed.