Comments (4)
leenoox
commented on July 3, 2024
1
I haven't been on on this and haven't reviewed all the changes since I
stopped contributing (life's too busy...) but in my case when one GPU
failed it locked all others as well, so tempcontrol might not be wrong in
reporting the problem for all gpu's.
If that's not the case (all gpu's locked) but tempcontrol errornesly
reports that all gpu's failed then you have to break the loop once the
condition was met (detected one failed gpu) and wait 60 seconds for
watchdog to reboot, and if it doesn't then execute reboot from tempcontrol
assuming watchdog died.
I believe I will have more free time in about a month and will rejoin the
development. I haven't reviewed the changes but according to all the mails
I get you guys are doing some serious coding. Rock on!
…On Tue, Mar 5, 2019, 7:42 AM Payam Nab, ***@***.***> wrote:
It looks like when tempcontrol query each GPU it passes the faulty GPU
error to all GPUs.
Bellow GPU0 with device handle 0000:01:00.0 is lost but warning error
appears on all GPU queries.
Tempcontrol log:
WARNING: Tue Mar 5 15:17:06 +0330 2019 - ***@***.*** Problem detected: nvidia-smi failed to query current status. Watchdog has 60 seconds to react, otherwise we will reboot!
========= SMI Query output with errors @[GPU0]:
Unable to determine the device handle for GPU 0000:01:00.0: GPU is lost. Reboot the system to recover this GPU
========= End of SMI query output
WARNING: Tue Mar 5 15:17:06 +0330 2019 - ***@***.*** Problem detected: nvidia-smi failed to query current status. Watchdog has 60 seconds to react, otherwise we will reboot!
========= SMI Query output with errors @[GPU1]:
Unable to determine the device handle for GPU 0000:01:00.0: GPU is lost. Reboot the system to recover this GPU
========= End of SMI query output
WARNING: Tue Mar 5 15:17:06 +0330 2019 - ***@***.*** Problem detected: nvidia-smi failed to query current status. Watchdog has 60 seconds to react, otherwise we will reboot!
========= SMI Query output with errors @[GPU2]:
Unable to determine the device handle for GPU 0000:01:00.0: GPU is lost. Reboot the system to recover this GPU
========= End of SMI query output
WARNING: Tue Mar 5 15:17:06 +0330 2019 - ***@***.*** Problem detected: nvidia-smi failed to query current status. Watchdog has 60 seconds to react, otherwise we will reboot!
========= SMI Query output with errors @[GPU3]:
Unable to determine the device handle for GPU 0000:01:00.0: GPU is lost. Reboot the system to recover this GPU
========= End of SMI query output
WatchDog log:
Tue Mar 5 15:16:57 +0330 2019 - [CRITICAL] - Lost GPU0 so restarting system. Found GPU's:
Unable to determine the device handle for GPU 0000:01:00.0: GPU is lost. Reboot the system to recover this GPU
Tue Mar 5 15:16:57 +0330 2019 - [CRITICAL] - reboot in 10 seconds
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#261>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ags0B3QTVkg0ugU-3W-apKw4MP9yurRVks5vTmYogaJpZM4behJ7>
.
from nvoc_by_fullzero_community_release.
LuKePicci
commented on July 3, 2024
@papampi that log seems correct. Where you see "SMI Query output with errors @[GPU2]:" it is the full query output, and it does not contain the only one related to the current gpu. It is correctly initiating a countdown to reboot for every GPU it has failed to query. There are some cases where the query fails for only a single GPU but returns valid results for the others and cases like this one where a major bus failure does prevent querying other devices since nvidia-smi is not giving other results apart from the error message. In this case, watchdog reacted before 60 seconds from the first failure, otherwise tempcontrol would have rebooted the rig by itself.
Therefore, I think it's working as expected. If you see from your logs another example of what happens when a single gpu query fails, you should also see outputs for other non-failing gpus between "SMI Query output with errors" as well.
@leenoox happy to see you back
from nvoc_by_fullzero_community_release.
papampi
commented on July 3, 2024
@leenoox
Cant wait to have you back ...
from nvoc_by_fullzero_community_release.
papampi
commented on July 3, 2024
@LuKePicci
When you get time can you see if we can make watchdog to behave like tempcontrol on problematic GPU to check it once more before rebooting the rig?
from nvoc_by_fullzero_community_release.
Related Issues (20)
- XMR-Stak don't start on 3main launching HOT 12
- NICE_CNV8 & NICE_CNHEAVY not displayed by WTM_SWITCHER HOT 10
- chose cuda version default HOT 1
- nvOC commands autocomplete HOT 3
- Algo and hashrate missing in minerinfo HOT 4
- Invalid default PHI2 miner HOT 1
- Can't add grin using gminer HOT 7
- Could you please add support for nicehash grin and beam using gminer 1.29? HOT 20
- Add support for multiple miner instances to segregate GPUs on single rigs HOT 7
- Please add support for cuckaroo31 using gminer 1.31 to mine grin31 HOT 11
- Could you please add bminer 14.3.1 support for cuckaroo29 and cuckatoo31 (pool and NH) HOT 13
- Aeternity option in gminer should be aeterity not cuckoo. Also please add f2pool as default AE pool HOT 1
- Disabled GPU not working correctly on PhoenixMiner and Claymore HOT 1
- Add support for NBminer 21.0 for grin etc algos HOT 3
- NICE_* address argument missing in miner command line HOT 4
- Common defaults should exploit COIN_MINER_OPTS where possible
- Add support RainbowMiner HOT 6
- nvoc 3.2 - not able to start ethminer HOT 9
- gtx 1660 super HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nvoc_by_fullzero_community_release.