Comments (6)
@jiarunying Thank you!问题找到了。看了下NM日志内存配置小了,这个NM内存怎么从2G突然到了6G,然后到降下来,没想到这么吃内存
from hbox.
同一台机器的两个Container不能通信么。worker一直卡着没有日志
from hbox.
正常情况下,worker运行结束后,am会向ps container发出作业完成信号,ps接收信号自行退出。此处没退出的原因需要排查日志分析。
同台机器可以启两个container,worker卡住需查看逻辑具体分析。是跑的demo出现的问题?
from hbox.
是跑Tensorflow demo出现的问题。。现在用了两台机器的NodeManager就是跑不通,其中一个是这个状态一直卡在这,个人感觉是因为另一个worker提前结束了,导致这个worker一直等着通信。
为什么Container会很快Complete呢?
from hbox.
而且demo是没问题的,我把demo.py 在同台机器,和不同机器跑都是没问题的
from hbox.
麻烦先排查下最早出现COMPLETE状态的container的退出原因(截图中是containerxxx_000003优先退出):是否存在RUNNING;如果存在RUNNING状态,则查看container日志的最后退出信息;若无,则查看NM端日志判断退出原因。因后边提示有task time out 日志,可能由于container中途被kill等原因导致的异常退出状态,与AM心跳通信超时。
from hbox.
Related Issues (20)
- 请问XLearning是否能在Ubuntu16.04下运行呢 HOT 2
- x-learning 在两个人同时执行demo时,最后报错 HOT 2
- 运行环境什么时候支持docker? HOT 3
- static file of xlearning can not be access in WebUI HOT 2
- Tensorboard not show any info HOT 1
- 大任务运行超过半小时后,分布式程序异常退出 HOT 1
- 环境变量超过最大长度该如何处理? HOT 2
- GPU support HOT 2
- [模型稳定性]同样的数据重复跑模型两次结果不一样,会存在模型预测结果不是确定的吗? HOT 1
- Unknown resource 'yarn.io/gpu' HOT 1
- Docker跑tf.estimastor hdfs问题 HOT 1
- 跑demo,出现java.lang.reflect.UndeclaredThrowableException HOT 1
- 请问,支持hadoop3.1么? HOT 2
- 请问XLearning支持spark任务提交吗? HOT 1
- 请问在提交Xlearning任务时出现java.lang.NoClassDefFoundError: org/mortbay/jetty/servlet/Context是什么问题 HOT 1
- There is a vulnerability in Hadoop 2.7.3 ,upgrade recommended
- 我运行 examples/tensorflow/run.sh 后一直显示Application report for application_1612431077961_007(state:ACCEPTED) HOT 2
- xlearning 运行时报错 HOT 1
- xlearning报错 HOT 2
- 请问下支持tensorflow2.0吗? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hbox.