Comments (4)
Snowdar
commented on August 26, 2024
Hi，如果你的特征放在机械硬盘上，这个问题就会出现，这是因为样本是动态随机加载的，机械硬盘的读写针头会不停的转动，耗费大量时间。如果你没有固态硬盘，那么你可以试试提前将egs打乱并另外并生成好一个特征的ark，然后关闭训练时的shuffle，这样读取egs时，机械硬盘为顺序整块读取，可以节省不少时间。另外，FastDataLoader提供的缓存机制也可以用上提升速度鲁棒性。
… On Sep 8, 2020, at 10:56 PM, hangxiu ***@***.***> wrote:
使用dataloader 多线程，但是多线程处于D状态，CPU读取数据慢，导师GPU util 为0， 请问这种情况怎么解决
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
from asv-subtools.
clockw
commented on August 26, 2024
hihi，我也遇到了同样的问题，多线程始终处于D状态，且data load的速度很慢，即使将shuffle关闭，并使用FastDataLoader，仍然需要很长的时间来load，导致训练速度极慢，请问你最后有解决这个问题嘛？或者还有什么更好的解决方法吗？谢谢！
使用dataloader 多线程，但是多线程处于D状态，CPU读取数据慢，导师GPU util 为0， 请问这种情况怎么解决
from asv-subtools.
cc-cherie
commented on August 26, 2024
请问这个问题是否有其他的解决办法，而且我发现在大数据量时，随着训练时间的增长，读取速度会越来越慢，请问有比较好的解决办法吗
from asv-subtools.
Snowdar
commented on August 26, 2024
Hi, 有类型的问题的朋友，可以提供一下以下信息吗？
1.特征存放位置的磁盘类型 （机械硬盘 ｜ 固态硬盘）
2.确认一下是加载egs映射列表慢还是训练时会慢
3.随着训练进行，是否有每个epoch后内存逐渐增加的问题
4.提供一下DataLoader的参数（线程数、pin_memory等）
from asv-subtools.
Related Issues (20)
- 多gpu运行 HOT 5
- 百度云盘的voxceleb recipe链接已经失效了，可以重新分享么？
- run Voxceleb Recipe [Speaker Recognition] HOT 3
- subtools/scoreSets.sh中191行特征提取错误数统计值errorNum HOT 1
- 多卡运行报错join() HOT 1
- the num_targets and the max label in train.egs.csv are not equal HOT 1
- Removing deprecated has_key function
- 多卡GPU运行失败 HOT 2
- 想完成一个说话人确认系统，应该怎么开始？ HOT 2
- Are there any plans to add the onnx conversion module to the model?
- rirs 和 musan 在哪里下载的？
- train.py,# Recover checkpoint 没有将loss的weight的保存，仅load模型，是不是有问题？ HOT 1
- 当使用ResNet模型进行迁移的问题 HOT 1
- subtools/pytorch/pipeline/export_jit_model.sh 导出模型报错 HOT 2
- runtime中的processor打分模块能否开源一下
- 是否准备提供onnx 导出功能的支持 HOT 2
- 发现了一个逻辑漏洞/found a bug
- online训练提示标签越界问题：Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"`
- subtools/pytorch/model/resnet_xvector.py文件152行多写了一个self x = self.self.cmvn_(x)
- extras/check_dependencies.sh: 13: function: not found HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from asv-subtools.