madynes / kadeploy Goto Github PK

Kadeploy is a scalable, efficient and reliable deployment system (cluster provisioning solution) for cluster and grid computing

License: Other

Ruby 84.73% Makefile 0.16% Shell 4.22% R 0.78% TeX 0.10% Perl 5.22% Pascal 0.01% Puppet 4.36% HTML 0.41%

kadeploy's People

Contributors

Stargazers

Watchers

Forkers

olbat ejeanvoi lnussbaum npf

kadeploy's Issues

give a way to retrieve environment files which as stored on server:///

if not on a NFS share mounted on the kadeploy client machine, how to get those files ?

Test issue

Hello there

kadeploy should chmod 1777 the tmp partition when used with -r

as of now, that was left to the postinstall

Internal Server Error in the user postinstall steps.

First guess: my postinstall file might be too small...

pneyron@fgrenoble:~/rootfs$ kadeploy3 -a centos7-kameleon.env -k -m genepi-23.grenoble.grid5000.fr -d
Deployment #D-f40656a5-ccec-4d55-bcf8-d6c048b42eaf started
Grab the key file /home/pneyron/.ssh/authorized_keys
Grab the tarball file /home/pneyron/rootfs/centos7.tgz
Grab the postinstall file /home/pneyron/rootfs/env-postinstall.tgz
Launching a deployment on genepi-23.grenoble.grid5000.fr
Performing a Deploy[SetDeploymentEnvUntrusted] step
  switch_pxe
  reboot
   * Performing a soft reboot on genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: ["ssh -q -F /etc/kadeploy3/keys/ssh_config [email protected] \"if [ -f /bin/systemctl ]; then nohup /bin/systemctl reboot -f >/dev/null & else nohup /sbin/reboot -f >/dev/null & fi\""]
[dbg] STATUS: 255
   * Performing a hard reboot on genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: ["/usr/local/bin/lanpower -c cycle -m genepi-23"]
[dbg] STATUS: 0
  wait_reboot
  send_key_in_deploy_env
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ cat - >>/root/.ssh/authorized_keys ] ; broadcast input file [ /var/cache/kadeploy/key_D_f40656a5_ccec_4d55_bcf8_d6c048b42eaf__home_pneyron__ssh_authorized_keys.data ]
[dbg] STATUS: 0
  create_partition_table
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ tmp=`mktemp` && chmod 755 ${tmp} && cat - > $tmp && KADEPLOY_CLUSTER="genepi" KADEPLOY_DEPLOY_PART="/dev/sda3" KADEPLOY_BLOCK_DEVICE="/dev/sda" KADEPLOY_DEPLOY_PART_NUM="3" KADEPLOY_SWAP_PART_NUM="1" KADEPLOY_PROD_PART_NUM="2" KADEPLOY_TMP_PART_NUM="5" KADEPLOY_PREPOST_EXTRACTION_DIR="/rambin" KADEPLOY_TMP_DIR="/tmp" KADEPLOY_ENV="centos7-kameleon" KADEPLOY_ENV_KERNEL="/boot/vmlinuz-3.10.0-514.10.2.el7.x86_64" KADEPLOY_ENV_INITRD="/boot/initramfs-3.10.0-514.10.2.el7.x86_64.img" KADEPLOY_ENV_KERNEL_PARAMS="console=tty0 console=ttyS1,115200n8" KADEPLOY_ENV_HYPERVISOR="" KADEPLOY_ENV_HYPERVISOR_PARAMS="" KADEPLOY_OS_KIND="linux" KADEPLOY_PART_TYPE="83" KADEPLOY_FS_TYPE="ext4" KADEPLOY_ENV_EXTRACTION_DIR="/mnt/dest"  ${tmp} ] ; broadcast input file [ /etc/kadeploy3/partitioning.sh ]
[dbg] STDOUT:   File: `/dev/sda'
[dbg] STDOUT:   Size: 0         	Blocks: 0          IO Block: 4096   block special file
[dbg] STDOUT: Device: 11h/17d	Inode: 7263        Links: 1     Device type: 8,0
[dbg] STDOUT: Access: (1660/brw-rw---T)  Uid: (    0/    root)   Gid: (    6/    disk)
[dbg] STDOUT: Access: 2017-03-09 16:05:23.599595927 +0000
[dbg] STDOUT: Modify: 2017-03-09 16:05:23.551595928 +0000
[dbg] STDOUT: Change: 2017-03-09 16:05:23.551595928 +0000
[dbg] STDOUT:  Birth: -
[dbg] STATUS: 0
  format_deploy_part
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ mkdir -p /mnt/dest; umount /dev/sda3 2>/dev/null; mkfs -t ext4 -b 4096 -O sparse_super,filetype,resize_inode,dir_index -q /dev/sda3 ]
[dbg] STATUS: 0
  mount_deploy_part
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ mount /dev/sda3 /mnt/dest ]
[dbg] STATUS: 0
  format_swap_part
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ mkswap /dev/sda1 ]
[dbg] STDOUT: Setting up swapspace version 1, size = 3905532 KiB
[dbg] STDOUT: no label, UUID=fabd92ad-2fe0-4de6-a108-338d1c8956a7
[dbg] STATUS: 0
End of step Deploy[SetDeploymentEnvUntrusted] after 102s
Performing a Deploy[BroadcastEnvKascade] step
  send_environment
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: kascade -i /var/cache/kadeploy/tarball_D_f40656a5_ccec_4d55_bcf8_d6c048b42eaf__home_pneyron_rootfs_centos7_tgz.data -n /tmp/kascade-nodefile20170309-29202-10l8wxs -O 'tar --selinux --xattrs --xattrs-include='*' -C /mnt/dest -xz' -S 'ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A' --ok /tmp/kascade-okfile20170309-29202-17l6ad4 --ko /tmp/kascade-kofile20170309-29202-sffmpx  -D taktuk -P '-d 0 -s' 
[dbg] STDERR: syck has been removed, psych is used instead
[dbg] STDERR: W, [2017-03-09T17:05:49.708289 #5513]  WARN -- : Connection failed on genepi-23.grenoble.grid5000.fr, retrying (1/10): Connection refused - connect(2) for "genepi-23.grenoble.grid5000.fr" port 29015
[dbg] STATUS: 0
   * Broadcast time: 11s
  manage_admin_post_install
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ tar --selinux --xattrs --xattrs-include='*' -C /rambin -xz ] ; broadcast input file [ /grid5000/preinstalls/preinstall-genepi.tgz ]
[dbg] STATUS: 0
  manage_user_post_install
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ tar --selinux --xattrs --xattrs-include='*' -C /rambin -xz ] ; broadcast input file [ /var/cache/kadeploy/postinstall_D_f40656a5_ccec_4d55_bcf8_d6c048b42eaf__home_pneyron_rootfs_env_postinstall_tgz.data ]
[dbg] STATUS: 0

---
Deployment status (251s):
  [BroadcastEnvKascade-manage_user_post_install] ~135s (OK)
     genepi-23.grenoble.grid5000.fr
---
[Internal Server Error]
---- NoMethodError ----
undefined method `join' for nil:NilClass
---- Stack trace ----
/usr/lib/ruby/vendor_ruby/kadeploy3/server/parallel_ops.rb:80:in `taktuk_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:293:in `block (2 levels) in parallel_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:260:in `parallel_op'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:286:in `block in parallel_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:301:in `call'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:301:in `parallel_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:2203:in `block in ms_manage_user_post_install'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:2192:in `each'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:2192:in `ms_manage_user_post_install'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:64:in `run'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/automata.rb:460:in `block (2 levels) in run_task'
---------------------

Is the ssh private key needed in the debirf deploy mini OS ?

Keys are:

addons/deploy_env_generation/debirf-jessie/kadeploy-deploy-kernel/kadeploy_specific/ssh/id_deploy
addons/deploy_env_generation/debirf-jessie/kadeploy-deploy-kernel/kadeploy_specific/ssh/id_deploy.pub

The public key is put in authorized_keys but I don't see anything regarding the use of the private key in the debirf deploy OS.

How does the chain work ? Is it using the ssh agent but no private key ?

Deploying on a different hard disk requires to specify the partition number

Hi,

Kadeploy lets the possibility to modify the hard disk on which it will deploys the environment (option "-b"). For instance, to deploy on /dev/sda instead of /dev/sdb (the default one, on my clusters), I execute kadeploy3 -m erods-ds-1 -e jessie-x64-nfs -b sda -k.

Such a command returns the following error:

Deployment #D-837a5766-6fe4-4faa-9c30-d0e947ed663c started
[Internal Server Error]
---- NameError ----
uninitialized constant Kadeploy::APIError::INVALID
---- Stack trace ----
/usr/lib/ruby/vendor_ruby/kadeploy3/server/workflow.rb:508:in `check_config'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/workflow.rb:144:in `load_config'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/automata.rb:165:in `initialize'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/workflow.rb:16:in `initialize'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/kaworkflow.rb:376:in `new'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/kaworkflow.rb:376:in `block (3 levels) in work_create'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/kaworkflow.rb:345:in `each_pair'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/kaworkflow.rb:345:in `block (2 levels) in work_create'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/kaworkflow.rb:335:in `synchronize'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/kaworkflow.rb:335:in `block in work_create'
---------------------

First it seems that the error is not correctly managed by Kadeploy. Besides, if we look closer on the source code (/usr/lib/ruby/vendor_ruby/kadeploy3/server/workflow.rb):

# Deploy on block device
if cexec.block_device and !cexec.block_device.empty? \
	and (!cexec.deploy_part or cexec.deploy_part.empty?)

	# Without a dd image
	unless cexec.environment.image[:kind] == ‘dd'
		error(APIError::INVALID,"You can only deploy directly on block device when using a dd image”)
	end

[…]

end

We can see that the if condition requires the use of the option "-p" together with the option "-b". Such a constraint is mentioned neither on the documentation, nor on the man page, nor on the error returned by Kadeploy. Besides, I do not understand such a constraint as I may want to modify the hard disk without modifying the partitioning.

Best regards,
Albin PETIT

Uploading file to TFTP/PXE does not work ?

pneyron@fgrenoble:~$ kareboot3 -x vmlinuz-3.16.0-4-amd64,initrd.img-3.16.0-4-amd64
No node is chosen

If I pass -m 'machine, the machine reboots...

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.