Comments (5)
加MP从实现角度肯定是没问题的,但是我们要考虑性价比的问题,在足够的人力和时间下,我们肯定也会把4D,5D都做出来。
我们现在任务是要梳理出一个优先级,把精力优先投入到最优利于用户使用和性能提升的技术上。
我觉得MP很鸡肋,我刚刚用DeepSpeed跑了一些实验结果
可以观察6B mp2的数据,和DP对比,性能下降很严重
【腾讯文档】PatrickStar_V100_profiling
https://docs.qq.com/sheet/DYWhORUNtREd1aGt0
from patrickstar.
模型并行这里我认同您的观点。Mesh-Tensorflow 这样的方案会比较难推给用户,可能短期内只需要有 Megatron-LM 的那种针对 transformer 的切分方式就够了。
不过最后的 chunk 那里的 1/n 我没太懂,是说每个参数都切分完放在各自进程内的 chunk 中,在使用的时候先 allgather 吗?
from patrickstar.
模型并行这里我认同您的观点。Mesh-Tensorflow 这样的方案会比较难推给用户,可能短期内只需要有 Megatron-LM 的那种针对 transformer 的切分方式就够了。
不过最后的 chunk 那里的 1/n 我没太懂,是说每个参数都切分完放在各自进程内的 chunk 中,在使用的时候先 allgather 吗?
是的,我觉得用zeroDP没必要用MP了。当然很多人不一定认同,还有很多人靠N-D并行吃饭
from patrickstar.
嗯,这个主要考虑需求吧。如果需求就是单纯的垒 transformer 最多能垒多少,那 MP 还是有优势的,毕竟只有 MP 可以拆分参数部分。但是如果考虑的是一般用户的应用,zeroDP 即插即用,要方便太多了
from patrickstar.
其实我觉得派大星和 MP 也不是矛盾的...用派大星劫持一下 Megatron-LM 的存储,不就相当于完成了 MP 了吗?所以我感觉和 deepspeed 中的 zero 一样,派大星的功能也是可以和 Megatrol-LM 这样的方案正交的。(可能额外需要把 allgather 改成 chunk 的 allgather 就是了)
from patrickstar.
Related Issues (20)
- Memory-centric tiling HOT 1
- Support both dynamic model data partition and static model data partition. HOT 1
- Polish memory and speed profiler.
- PatrickStar's Performance in Models Like GANs HOT 2
- Support NVMe HOT 1
- 运行报错 HOT 1
- Optimize chunk allocate and release HOT 2
- Proposal: overlap NVMe read and write with computing. HOT 2
- Skipping ADAM in warmup affects the overall performance.
- Support communication config before training
- Search the best chunk size. HOT 1
- Accelerate Chunk List Construction Speed. HOT 1
- support using PatrickStar on MegatronDeepSpeed? HOT 3
- Error when install under python3.6 HOT 1
- FP32ChunkReadBuffer throw errors for vit training.
- 希望能够保持特定层的 weight 仍为 float32 HOT 5
- A major refactor to sacrifice some performance for flexiblity and simplicity HOT 1
- RuntimeError: chunk move failed. HOT 3
- install issue HOT 2
- Hi,What is the offset do?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from patrickstar.