Comments (2)
I always assumed this was due to heavy kvs usage, especially the multitude of commits
that happen with each line of I/O. However, I realize I never actually characterized
the performance of this proof-of-concept code, and I guess we should probably do that to
understand how future remote execution implementations can and should leverage the kvs.
In my testing I find that using -o stdio-delay-commit
pretty much erases the performance
degradation. However, I don't really understand why performance plateaus instead of
getting continually worse. Also, sleeping for a bit between runs seems to help too.
There is still a bit of a performance impact with subsequent jobs when using stdio-delay-commit
,
but much less so than without:
~/git/flux-core.git/b/src/cmd $ for i in `seq 1 10`; do ./flux wreckrun -N8 -t16 hostname 2>&1 | grep 'State = complete'; done
wreckrun: 0.778s: State = complete
wreckrun: 5.782s: State = complete
wreckrun: 6.723s: State = complete
wreckrun: 9.053s: State = complete
wreckrun: 9.353s: State = complete
wreckrun: 8.972s: State = complete
wreckrun: 7.814s: State = complete
wreckrun: 8.843s: State = complete
wreckrun: 9.228s: State = complete
wreckrun: 7.737s: State = complete
$ ~/git/flux-core.git/b/src/cmd$ for i in `seq 1 10`; do ./flux wreckrun -o stdio-delay-commit -N8 -t16 hostname 2>&1 | grep 'State = complete'; done
wreckrun: 0.222s: State = complete
wreckrun: 0.226s: State = complete
wreckrun: 0.228s: State = complete
wreckrun: 0.473s: State = complete
wreckrun: 0.472s: State = complete
wreckrun: 0.557s: State = complete
wreckrun: 0.469s: State = complete
wreckrun: 0.512s: State = complete
wreckrun: 0.513s: State = complete
wreckrun: 0.502s: State = complete
from flux-core.
As with the hang, this specific bug seems to have been addressed.
from flux-core.
Related Issues (20)
- add a --config-path=PATH option to flux config get HOT 2
- job-exec: add config reload callback
- add tooling for poking at sdexec transient units
- sdexec: transient units may not be getting cleaned up properly
- flux-perilog-run exits silently with failure when one or more ranks are not online HOT 1
- cron: race may cause cron jobs to never run again
- job manager: track average and maximum job wait time HOT 1
- mpi: multiple brokers per node with multiple MPI tasks per node hangs with mvapich2-2.3.7-intel HOT 2
- shell: slow job termination when `exit-on-error` triggered
- rank 0 broker crash on elcapi HOT 4
- Allow --dependency on job name HOT 25
- job-exec: possible scaling issue managing many job shells HOT 37
- shell/oom: error handling is sub par
- do not use `flux_cmd_setcwd()` unless necessary
- shell segfault when evlog kvs commit times out HOT 1
- flux-resource: add `-q, --queue` and/or `-p, --properties` option to `flux resource list` HOT 1
- add output mode in `flux job taskmap` to map hostnames to taskids
- need to clean old resource.eventlogs
- limit cpus seen by hwloc in Flux jobs HOT 18
- libsubprocess: flux_subprocess_read() could return 0 despite not EOF HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flux-core.