cloudstack-extras / knife-cloudstack Goto Github PK

It would be great if you could use/borrow code from chef/knife where it can now use csshX to login to multiple systems at once, using Mac terminal. Could you use this logic with the above command to create a "stack" of nodes in parallel terminal windows making it exponentially faster to standup a stack/cluster. With the latest knife you can run:

knife ssh "role:blah AND chef_environment:blah" -a ipaddress csshx

It would be awesome if you could do this with the stack command, something like:

knife cs stack create blah.json csshx

and it launches each of the nodes in a separate window with csshX.

The next stage to the holy grail of massive cluster/single command automation.

It would be nice to be able to specify default run lists when creating servers, something along the lines of the following gist.

https://gist.github.com/3649331

When creating a server on cloudstack which we don't want to have bootstrapped we use --no-bootstrap switch but knife cs is still trying to connect to the ssh daemon. So it seems this switch is being ignored.

I've installed chefdk 0.3.5 and then installed knife-cloudstack using the chef environment:

> chef gem install knife-cloudstack
Successfully installed knife-cloudstack-0.0.17
Parsing documentation for knife-cloudstack-0.0.17
Installing ri documentation for knife-cloudstack-0.0.17
Done installing documentation for knife-cloudstack after 2 seconds
1 gem installed

Then tried to run it and got this stackstrace.

> knife cs
/opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:2099:in `raise_if_conflicts': Unable to activate chef-11.6.2, because mixlib-config-2.1.0 conflicts with mixlib-config (>= 1.1.2, ~> 1.1), ohai-7.4.0 conflicts with ohai (< 7.0.0, >= 0.6.0), json-1.8.1 conflicts with json (<= 1.7.7, >= 1.4.4), net-ssh-multi-1.2.0 conflicts with net-ssh-multi (~> 1.1.0) (Gem::ConflictError)
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1282:in `activate'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:90:in `block in require'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:89:in `each'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:89:in `require'
    from /Users/mferreira/.chefdk/gem/ruby/2.1.0/gems/knife-cloudstack-0.0.17/lib/chef/knife/cs_baselist.rb:19:in `<top (required)>'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:128:in `require'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:39:in `require'
    from /Users/mferreira/.chefdk/gem/ruby/2.1.0/gems/knife-cloudstack-0.0.17/lib/chef/knife/cs_aag_list.rb:20:in `<top (required)>'
    from /opt/chefdk/embedded/apps/chef/lib/chef/knife/core/subcommand_loader.rb:37:in `load'
    from /opt/chefdk/embedded/apps/chef/lib/chef/knife/core/subcommand_loader.rb:37:in `block in load_commands'
    from /opt/chefdk/embedded/apps/chef/lib/chef/knife/core/subcommand_loader.rb:37:in `each'
    from /opt/chefdk/embedded/apps/chef/lib/chef/knife/core/subcommand_loader.rb:37:in `load_commands'
    from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:120:in `load_commands'
    from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:168:in `run'
    from /opt/chefdk/embedded/apps/chef/lib/chef/application/knife.rb:139:in `run'
    from /opt/chefdk/embedded/apps/chef/bin/knife:25:in `<top (required)>'
    from /opt/chefdk/bin/knife:33:in `load'
    from /opt/chefdk/bin/knife:33:in `<main>'

when knife.rb setting default api key/secret key of cloudstack user's, "knife cs stack create" picks them up and ignores CLI options. Other knife cs commands user CLI options before default setting in knife.rb. why? I suspect it's a bug...?

Changes to allow:

• stack delete uses correct fqdn-less node name when deleting chef node and client
• new option to skip existing servers already created when creating a stack - makes for better usability when dealing with large numbers of servers
• cosmetic changes to output

The --[no]-public-ip option isn't available in knife cs stack create causing the API to return "errorcode": 431, "errortext": "Can't specify network Ids in Basic zone", similar to issue #1.

I'm trying to use the "knife cs stack create" command with a json file but I want to specify a unique template-file (bootstrap file) and I can't seem to find a way to do it. I notice that it's trying to run the default one and when I pass "--template-file bootstrap/ubuntu10.04-ruby193.erb" It fails with a:

Error: invalid option: --template-file

If you are using virtual networking, knife cs server create will fail if the virtual router has not been initialized, with the error below. The workaround is to create a vm instance via the webui. This will initialize the router. After that knife cs will work properly.

$ knife cs server create vrweb0 -S M -x foo -P bar-r 'role[base]' -VV
Waiting for server/Library/Ruby/Gems/1.8/gems/knife-cloudstack-0.0.11/lib/knife-cloudstack/connection.rb:157:in create_server': undefined method[]' for nil:NilClass (NoMethodError)
from /Library/Ruby/Gems/1.8/gems/knife-cloudstack-0.0.11/lib/knife-cloudstack/connection.rb:156:in map' from /Library/Ruby/Gems/1.8/gems/knife-cloudstack-0.0.11/lib/knife-cloudstack/connection.rb:156:increate_server'
from /Library/Ruby/Gems/1.8/gems/knife-cloudstack-0.0.11/lib/chef/knife/cs_server_create.rb:174:in run' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife.rb:391:inrun_with_pretty_exceptions'
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife.rb:166:in run' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/application/knife.rb:128:inrun'
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/bin/knife:25
from /usr/bin/knife:19:in `load'
from /usr/bin/knife:19

Add a gem dependency check with https://gemnasium.com and display the badge on the readme file. This way we can show that the knife plugin has up-to-date dependencies.

The option is already set by a dependency, in the knife-windows gem: https://github.com/chef/knife-windows/blob/master/lib/chef/knife/winrm_base.rb#L75-L79, so the instructions in the README don't work.

I think the short form option should be removed, and the README updated. Should I create a PR for that? Thanks in advance

I thought It would be very useful to be able to pass the attributes when creating a new server.

Both "knife cs server list" and "knife cs hosts" throw out that error.
cloudstack version 2.2.14
chef version 0.10.8
knife-cloudstack version 0.0.11

I've noticed that if one of the VMs breaks or doesn't complete and you try to run the "knife cs stack create" command again, it'll complain that the first node already exists and won't continue.

It would be great if it could just acknowledge that to the user but continue on until it finds a node in the json file that doesn't exist and starts launching it... and then continue on with the rest of the "stack". If you want to launch a large group of machines and there's a problem with just one of them, you're pretty much stuck either having to do the rest manually or destroying the ones that completed and running the stack create command again.

As it is, this module does not allow a user to generate keys or apply those keys when provisioning a new server.

Creating a key seems like a straight forward task and if that private key is saved to ~/.ssh or ~/.chef, it can be referenced from the knife.rb by keypair name and file location.

The keypair can be used when creating a server by adding 'keypair' to the API call. The knife bootstrap command already supports an identify file with the -i parameter.

Joining all of these, a user would be able to provision and bootstrap a node with a passwordless login.

Since networks are stored in stack definitions as a space/comma separated string, it's not possible to specify a network with a space in the name because this will be split into multiple entries. This data is coming from JSON, so why not use a JSON Array?

this is the outout of the command that I get on CloudStack 4.1.0

knife cs keypair create  --name test  --print-after
Creating SSH Keypair: test

Name:         test
Fingerprint:  21:66:c3:9c:2a:ce:98:0c:f6:30:30:05:2c:38:e8:fe
PrivateKey:

As for now it create a VM based on the template disk offering, it should be possible to change the offering using the diskofferingid
Thanks
Luca

The -T (cloudstack template) option on the command line is overridden by setting this option in knife.rb. It should work the other way around. Explicit command line options should override configs from the knife config file.

Using knife cs server list always returns the address from the guest network for the virtual machine and not the actual public IP used by the virtual router. When trying knife cs server create it creates and starts the VM as expected but reports the guest network IP as the public IP and hangs while waiting for SSH. Is there something I'm possibly doing wrong or is there issues with advanced networking with source NAT for this knife command?

This part of connection.rb looks as if it should be finding the public IP and forwarding rule for virtual networks but it's not working in my experience:

def get_server_public_ip(server, cached_rules=nil)
  return nil unless server

  # find the public ip
  nic = get_server_default_nic(server) || {}
  if nic['type'] == 'Virtual' then
    ssh_rule = get_ssh_port_forwarding_rule(server, cached_rules)
    ssh_rule ? ssh_rule['ipaddress'] : nil
  else
    nic['ipaddress']
  end
end

It always list the public IP address as the 10.1.1.0/24 address from the guest network.

/usr/lib/ruby/gems/1.8/gems/chef-0.10.10/lib/chef/knife

line 149:
session_opts[:password] = config[:ssh_password] if config[:ssh_password]
should change to
session_opts[:password] = Chef::Config[:knife][:ssh_password] || config[:ssh_password]

Otherwise can't pass the password to ssh

It seems like Cloudstack doesn't like when the api tries to list a network when the zone is in basic mode. Can you please add an option to not tell the knife plugin to specify a network for both the server create and cluster create functions?

thanks
Chris

sample command:

knife cs server create fromchef1 -T col-rhel5.6-x64-v0.5.1 -S "Small Instance" --no-bootstrap -x xxx -P xxx

Waiting for serverError 431:
{
"deployvirtualmachineresponse": {
"errorcode": 431,
"errortext": "Can't specify network Ids in Basic zone"
}
}
URL: http://XXXXXXXXXXXXXXXXXXXXXX:8080/client/api?apiKey=XXXXXXXXXXXXXXXXXXXXXX&command=deployVirtualMachine&name=fromchef1&networkids=3eab1fb9-4efe-43fe-83a7-9c641ee7470c&response=json&serviceOfferingId=76b352d8-5ad2-4199-a4ca-39fd18664912&templateId=7e36edf6-f928-4fa4-b03b-cc6d3b2117ce&zoneId=303dc4d7-afc9-4508-954c-fcf15fe457c9&signature=wG1Ww1HPlEYCgiIucJwZD5FTbHI%3D

The user/password/sshkey options are normally required for knife bootstrap and knife cs server create but should not be required if --no-bootstrap is used.

For instance,

1. vm name "sample10" is deployed.
2. "cs sever create sample1"
3. knife cloudstack informs the error that Error: Server 'sample1' already exists.

According to verbose output, "listVirtualMachines name=sample1" returns sample10 vm information.
We use cloudstack 4.2.x. It may be a problem of cloudstack but knife cloudstack should strictly check the response of listVirtaulMachines....

When locating a template the following values are allowed by cloudstack:

"featured", "self", "selfexecutable","sharedexecutable","executable", and "community"

Currently it's hardcoded to 'executable' meaning templates in any other section are not found. For example, all templates I uploaded to my cloud provider are in 'self'.

I want to manage the private cloudstack to use the 'kinfe-cloudstack' plugin.
but, when cs command run, it was generated the error message:

• /net/http.rb:799:in `connect': SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol (OpenSSL::SSL::SSLError)

so, I added 'http.ssl_version = :SSLv2' in the 'send-request' method of the ''/knife-cloudstack/conection.rb".
but, when cs command run, it was generated the error message:

• /net/http.rb:799:in `connect': SSL_set_tlsext_host_name: (OpenSSL::SSL::SSLError)

i don'k know.. what is problems..?

I have configured as follow:

1. I installed the 'kinfe-cloudstack' plugin on my chef workstation.
   (gem install knife-cloudstack)
2. and '~/.chef/knife.rb' added :
   knife[:cloudstack_url] = 'http://my.cloud.net:8080/client/api'
   knife[:cloudstack_api_key] = 'xxxxxxx'
   knife[:cloudstack_secret_key] = 'xxxxxx'
3. Enabling SSL in the CloudStack UI (https://cwiki.apache.org/CLOUDSTACK/enabling-ssl-in-the-cloudstack-ui.html)

what are problems?
help me.. please...

When using basic networking mode nothing shows up when running knife cs network list. Additionally this causes problems when running knife cs server delete as a FQDN cannot be attained to successfully remove the node & client from Chef after the instance is destroyed.

$ knife cs network list
Name     Type     Default  Shared   Gateway  Netmask
$

When you launch a single instance like:

knife cs server create intmhva1 -N intmhva1 -E integration -r 'role[mh_video_api]' -S "Large (Local)" -T "MHIMG-20120409" --template-file bootstrap/ubuntu10.04-ruby193.erb

the name in "knife cs server list" and the name in "knife node list" and knife client list" are consistent

But when I try to create a stack with the json file with the following format:

$ cat scripts/intmhv.json
{
"name": "intmhv",
"description": "Frontend and Load Balancers",
"version": "0.1",
"environment": "integration",
"servers": [
{
"name": "intmhv1 intmhv2",
"description": "FE nodes",
....

depending on how the administrator configures the default TLD, the names will be inconsistent. I've tried adding node_names and chef_node_names like this:

$ cat scripts/intmhv.json
{
"name": "intmhv",
"description": "Frontend and Load Balancers",
"version": "0.1",
"environment": "integration",
"servers": [
{
"name": "intmhv1 intmhv2",
"chef_node_name": "intmhv1 intmhv2",
"description": "FE nodes",
....

but it doesn't work. What can I pass in the json file to make sure the chef and cloudstack names are consistent, as if I was launching them individually?

$ knife cs server delete intmhvsrchs1 -VV
DEBUG: Using configuration from /.chef/knife.rb

Name: intmhvsrchs1
Public IP:
Service: Large
Template: IMG20120409
Domain: DEV
Zone: DEV1
State: Running

Do you really want to delete this server? (Y/N) y
Waiting for deletion.........
Deleted server intmhvsrchs1
Do you want to delete the chef node and client '? (Y/N) y
DEBUG: Signing the request as
DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations//nodes/
/Users/dfuentes/.rvm/gems//gems/knife-cloudstack-0.0.11/lib/chef/knife/cs_server_delete.rb:130:in delete_node': undefined methoddestroy' for #Hash:0x007fdf958c7630 (NoMethodError)
from /Users/dfuentes/.rvm/gems//gems/knife-cloudstack-0.0.11/lib/chef/knife/cs_server_delete.rb:85:in block in run' from /Users/dfuentes/.rvm/gems/<ruby>//gems/knife-cloudstack-0.0.11/lib/chef/knife/cs_server_delete.rb:51:ineach'
from /Users/dfuentes/.rvm/gems///gems/knife-cloudstack-0.0.11/lib/chef/knife/cs_server_delete.rb:51:in run' from /Users/dfuentes/.rvm/gems/<ruby>//gems/chef-0.10.8/lib/chef/knife.rb:391:inrun_with_pretty_exceptions'
from /Users/dfuentes/.rvm/gems///gems/chef-0.10.8/lib/chef/knife.rb:166:in run' from /Users/dfuentes/.rvm/gems/<ruby>//gems/chef-0.10.8/lib/chef/application/knife.rb:128:inrun'
from /Users/dfuentes/.rvm/gems///gems/chef-0.10.8/bin/knife:25:in <top (required)>' from /Users/dfuentes/.rvm/gems/<ruby>//bin/knife:19:inload'
from /Users/dfuentes/.rvm/gems///bin/knife:19:in `

See https://issues.apache.org/jira/browse/CLOUDSTACK-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel for more details.

I'm probably the one that should fix this.

It would be nice if you didn't have to specify a hostname (and let cloudstack auto assign one)