Comments (5)
transcription of timing data...
some timings: Vagrant 2.2.9, Virtualbox 6.1.6, underlying platform; Fedora30: 5.6.13-100
VirtualBox VMs file system: xfs over luks, underlying SSD/nvme
tarballs file system: xfs, underlying SSD/nvme
vcfs: 9m, ingest/createrepo repos.tgz, yum install/timeouts, selinux homedirs
vcsvc: 3m, yum install/timeouts, rsyslog selinux
vcbuild: 11m, yum (epel/priorities) install/timeouts, slurm build - if triggered, prereq timeouts
vcdb: 7m, yum (epel/priorities) install/timeouts, mysql (community mysql repo?) install, prereq timeouts
vcaltdb: 9m, yum (epel/priorities?) install/timeouts, mysql (community mysql repo?) install, prereq timeouts
vcsched: 6m, yum (epel/priorities?) install/timeouts, prereq timeouts
vc1: 13m, yum (epel/priorities?) install/timeouts, prereq timeouts
vc2: 4m, yum (epel/priorities?) install timeouts, prereq timeouts
vclogin: 22m, slurm test jobs, slurm db/config, yum (epel/priorities?) install timeouts
vxsched: 3m, yum install (epel/priorities?) timeouts
vx1: 5m, yum install (epel/priorities?) timeouts
vx2: 4m, yum install (epel/priorities?) timeouts
vxlogin: 10m, slurm test jobs, slurm db/config, yum (epel/priorities?) install timetouts
muon/centos underlying similar, slower: VirtualBox VMs & tarballs: zfs, raidz1 over HDD
So, first-cut:
- cache more from epel into local repositories, attempting to remove it from most/all nodes,
- find another mechanism besides yum-plugin-prorities to force local repo first, then remote fallback,
- tune prereq timeouts
- all else, such as vclogin, slurm-test-jobs, selinux
from hpc-collab.
VMtouch, to lock memory pages in the host: https://github.com/hoytech/vmtouch
User-level NFS server, replacing vboxsf
preliminary timings don't indicate much of a gain, implying the problem isn't host I/O, but rather host to VM I/O or in-VM I/O.
Cpu load is not high when this is occurring, implying that it is not trashing on the host system. (in either the host or guests)
from hpc-collab.
Contacting the epel repository is a source of delay in node provisioning. This is a wait for network I/O, rather than any real productive work. This may be best addressed by a snapshot-checkpoint-suspend/resume cycle.
This has been addressed by careful caching of RPMs to avoid epel and yum search timeouts, especially in the early RPM installation step.
from hpc-collab.
compute nodes may be unprovisioned and reprovisioned safely, separate from other nodes.
from hpc-collab.
estale monitoring code added, allowing most nodes (except vcdb) to be reprovisioned. In particular, vcfs may disappear and reappear, much like a traditional NFS server
from hpc-collab.
Related Issues (20)
- create backup slurmdbd instance, either on vcaltdb or separately
- incorporate a queue visualizer: (1) http://paulbourke.net/miscellaneous/gqstats/ (2) https://github.com/hnieto/TACC_SuperComputer_Queue HOT 1
- integrate collectd slurm plugin to generate data for #189 visualization (https://github.com/hpc/hpc-collab/issues/189)
- Add vz virtualization provider, enabler to using proxmox as a turnkey cluster env. HOT 4
- add 'synchome' option (removing old /vagrant fstab mount) to copy data from within the cluster back to the host
- create a simple recipe, appropriate to validate a slurm configuration settings (slurm.conf, cgroup.conf, slurmdbd.conf)
- refactor early RPMS into Makefile/ingest manifests HOT 1
- in-cluster OS alternatives to CentOS 7: Oracle Linux, Cloud Linux, Springdale Linux, Fedora, Debian, others?
- convenient mechanism/recipe/tool to add users to an existing cluster *or* make reprovisioning so lightweight & fast that it is convenient to reprovision with a new user set (#23, #9) HOT 1
- federated cluster recipe, extending & expanding vc+vx to fc+fx
- cluster recipe to include burst-to-external-cloud (Google, AWS) compute node(s)
- new cluster recipe: multiple slurmdbd instances, for linked multicluster w/QOS overlap HOT 1
- create and populate a repo "local-cache" with contents of cfg/<cluster>/<node>/rpm/{early,install}/*.rpm HOT 1
- openhub project URL: https://www.openhub.net/p/hpc-collab
- add BeeGFS to vcbuild, clients to compute and login nodes, possibly sched nodes
- accommodate multinode HPC allocation rather than single-node VM instantiation
- transition project to using OpenStack+RDO rather than vagrant+{virtualbox,kvm}
- build fails on AArch64, Fedora 33
- add nested virtualization optimizations like 'choom', 'vmtouch', possibly 'renice'
- when im trying to srun in cluster i am getting issue can any one help to rectify the issue HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hpc-collab.