Comments (7)
Short answer, use flux submit
instead of flux mini submit
.
flux mini
was deprecated in v0.48.0 (2023-03-7) and dropped in v0.59.0 (2024-02-06) in favor of flux submit
, flux run
, flux alloc
, etc.
from flux-core.
Great, that worked, thanks! However, I'm having issues where Slurm is killing my job after the first 720 tasks are run. I'm getting these messages: broker.err[3]: quartz405 (rank 7) transitioning to LOST due to EHOSTUNREACH error on send
, and at then at the end I get this:
srun: error: Node failure on quartz405
slurmstepd: error: *** STEP 2483483.0 ON quartz116 CANCELLED AT 2024-03-27T12:32:44 DUE TO NODE FAILURE, SEE SLURMCTLD LOG FOR DETAILS ***
slurmstepd: error: *** JOB 2483483 ON quartz116 CANCELLED AT 2024-03-27T12:32:44 DUE TO NODE FAILURE, SEE SLURMCTLD LOG FOR DETAILS ***
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
Mar 27 12:32:44.534897 broker.err[0]: rc2.0: ./runScript.sh 0 Hangup (rc=129) 6988.9s
from flux-core.
Any ideas on why this is happening? I tried to search for the SLURMCTLD
log file, but I couldn't find it
from flux-core.
It looks like quartz405 crashed or became unresponsive according to Slurm. If there wasn't an actual node failure, another reason this could happen is that the node is run out of memory.
The slurmctld log file is usually on the management node and is viewable by sysadmins only, not sure why Slurm is telling the user to look there. If nothing else is evident, you could try contacting the local hotline to ask what happened to that node.
from flux-core.
Got it, thank you!
from flux-core.
BTW, I'll just mention here since I noticed this in your script:
Usage such as:
for proc in {0..719}
do
JOBIDS="$JOBIDS $(flux submit -n1 --output=result.${proc}.out command input.${proc}.in)"
done
wait_all "$JOBIDS"
Can now be much more efficiently submitted and waited for in a single submit command with
flux submit --wait --cc=0-719 -n1 --output=result.{cc}.out command input.{cc}.in
The --cc
option submits multiples of the same job, with {cc}
substituted with the index in each submission.
The --wait
option in flux submit
waits for all submitted jobs to complete before returning.
I'm not suggesting you change your current script if it is working fine now, but this is more a FYI and instructional for others that may stumble across this issue report. flux submit -cc
will be many times faster at submitting jobs than multiple flux submit
calls in a loop.
Thanks!
from flux-core.
Assuming this question is answered, I'll go ahead and close the issue. Feel free to reopen if there's anything else.
from flux-core.
Related Issues (20)
- python: `!D` output conversion specifier should return an empty string for `0.0` HOT 1
- update usage/docs for cmds with options `-L, --color[=WHEN]` to indicate that `-L` can't take the `WHEN` argument and `=` is required
- sched.alloc-response: id=fF1SDRy already allocated HOT 2
- pty: flux pty code drops ctrl-<space> HOT 1
- broker proctitle is truncated at 16 bytes HOT 2
- flux-shutdown: need option to force fast shutdown HOT 2
- `flux job wait-event --match-context=KEY=VAL` matches any `KEY` when `VAL` matches
- idea: add sync option to job manager journal
- resource: `resource.status` RPC is very slow on el cap HOT 9
- `flux resource list` performance issue when queues are configured
- segfault on rank 0 broker of elcapi HOT 1
- flux-resource: exponential time complexity processing drained ranks
- properly report task exit status when signaled in `flux job attach` and other utilities
- Missing test_must_fail function in Flux testing environment HOT 19
- broker spending a lot of time in `child_lookup()` HOT 2
- broker crash in `prepare_sched_status_payload` HOT 2
- support ANY with simple job dependencies
- broker version operability constraints are too tight
- job-manager: a job must wait for scheduler to respond to a canceled allocation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flux-core.