Comments (7)
I gave a test for my use case, seems all works fine! Thanks a lot for the help, I close the issue as resolved.
from flux-core.
For multi-user, you may be able to edit the systemd unit file to set up resource control e.g. AllowedCPUs
and MemoryMax
, thus restricting flux to a given set of cores and memory using cgroups.
The admin guide describes how to set up multi-user Flux which is a little more involved than what @grondo is describing, so his suggestion may be a better starting point.
from flux-core.
There is no real downside except that you will be oversubscribing resources.
from flux-core.
Flux will detect the resources available to it at startup (using hwloc), so an easy way to run flux on a subset of resources is to launch flux start
under taskset(1) or hwloc-bind(1).
grondo@pi3:~$ flux start flux resource list -s free -o rlist
STATE NNODES NCORES NGPUS LIST
free 1 8 0 rank0/core[0-7]
grondo@pi3:~$ hwloc-bind core:0-3 flux start flux resource list -s free -o rlist
STATE NNODES NCORES NGPUS LIST
free 1 4 0 rank0/core[0-3]
Even if you specified cores 4-7
here, Flux will renumber the cores to 0-3
, however when launching job tasks they should still be constrained to real cores 4-7
since Flux will index the cores relative to those core it has access.
If you want to run a multi-user instance of Flux, then that would be a slightly different setup.
from flux-core.
Cool, thanks a lot for the help! I think that what I need, the single user solution is fit for my use case for the moment.
I notice with --test-size <num>
can create more brokers than CPUs, is that possible let's say using 2 CPUs in total but can further separate it to run 4 jobs with each one using 0.5 CPUs? (Sorry if I confused with the terminology)
from flux-core.
Running multiple brokers on the same node will oversubscribe resources. This is mainly used for testing, but some users find this practice useful when performance is not a concern, e.g. for CI jobs. Flux uses this in its own testsuite to simulate a multi-node instance on a single host.
To specifically answer your question:
let's say using 2 CPUs in total but can further separate it to run 4 jobs with each one using 0.5 CPUs? (Sorry if I confused with the terminology)
Yes, if you run 2 brokers on 2 cores, your Flux "instance" will see 4 cores and may schedule up to two 'tasks' per core.
from flux-core.
So I need to use --test-size=2
I assume, it says "a test instance" is there any downside of using it, for my case to use local machine as a cluster I think it cover the use case?
from flux-core.
Related Issues (20)
- startlog: need to initialize flag enum to a number HOT 1
- content-sqlite: preallocate database space HOT 26
- some JSON implementations may not be able to handle the full range of job IDs encoded as JSON numbers HOT 2
- shell: increase `output.batch-timeout` default for very large jobs HOT 2
- housekeeping drain message is not informative after timeout HOT 1
- python: need easy way to wait until a subinstance is ready to accept jobs HOT 1
- housekeeping doesn't release resource for a large job HOT 11
- content: flush data if backing store is loaded at a later time HOT 8
- t2406-job-exec-cleanup Racy sub-test HOT 1
- idea: vacuum sqlite on ENOSPC errors during teardown HOT 2
- flux overlay errors does not display version mismatch information
- Suggestion: ability to specify a node range in flux jobs HOT 1
- satisfiability bug: specifying a queue and a constraint matching another queue hangs HOT 1
- throughput.py test: incorrect number of tests submitted in status HOT 10
- job is terminated with `scheduler-restart` exception on Fluxion module reload
- not ok 17 - attach: reports job shell Killed if job shell is killed
- Interested in elastic deployments on Slurm HOT 7
- job-manager: job update feasibility check topic string should be configurable HOT 2
- `flux perilog-run` is not scalable
- job-manager: jobs with alloc-bypass flag appear to still generate a free request to scheduler HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flux-core.