Thanks!
-Dave

When running chef-client on jenkins-master-01 we get the following erro (see ticket 140815-11214):

Also should the private key thats installed have a \n at the end?

Error executing action `create` on resource 'jenkins_private_key_credentials[jenkins]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /usr/lib/jvm/java/bin/java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 groovy /tmp/groovy20140817-15019-10im3k2 ----
STDOUT:
STDERR: hudson.security.AccessDeniedException2: anonymous is missing the Overall/RunScripts permission
    at hudson.security.ACL.checkPermission(ACL.java:55)
    at hudson.model.Node.checkPermission(Node.java:417)
    at hudson.cli.GroovyCommand.run(GroovyCommand.java:74)
    at hudson.cli.CLICommand.main(CLICommand.java:234)
    at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:92)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:309)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:290)
    at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:249)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:328)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at hudson.cli.CliManagerImpl$1.call(CliManagerImpl.java:63)
    at hudson.remoting.InterceptingExecutorService$2.call(InterceptingExecutorService.java:95)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
---- End output of /usr/lib/jvm/java/bin/java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 groovy /tmp/groovy20140817-15019-10im3k2 ----
Ran /usr/lib/jvm/java/bin/java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 groovy /tmp/groovy20140817-15019-10im3k2 returned 255

Cookbook Trace:
---------------
/var/chef/cache/cookbooks/jenkins/libraries/_executor.rb:79:in `execute!'
/var/chef/cache/cookbooks/jenkins/libraries/_executor.rb:109:in `groovy!'
/var/chef/cache/cookbooks/jenkins/libraries/credentials.rb:205:in `current_credentials'
/var/chef/cache/cookbooks/jenkins/libraries/credentials_private_key.rb:117:in `current_credentials'
/var/chef/cache/cookbooks/jenkins/libraries/credentials.rb:79:in `load_current_resource'
/var/chef/cache/cookbooks/jenkins/libraries/credentials_private_key.rb:69:in `load_current_resource'

Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/jenkinsstack/recipes/master.rb

 35: jenkins_private_key_credentials node['jenkins']['master']['user'] do
 36:   username node['jenkins']['master']['user']
 37:   description 'Jenkins Slave SSH Key'
 38:   private_key s_private_key
 39: end
 40:

Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/jenkinsstack/recipes/master.rb:35:in `from_file'

jenkins_private_key_credentials("jenkins") do
  action :create
  retries 0
  retry_delay 2
  guard_interpreter :default
  cookbook_name "jenkinsstack"
  recipe_name "master"
  username "jenkins"
  description "Jenkins Slave SSH Key"
  private_key "-----BEGIN RSA PRIVATE KEY-----\nREMOVED=\n-----END RSA PRIVATE KEY-----\n"
end

Running handlers:
[2014-08-17T20:31:55+00:00] ERROR: Running exception handlers
Running handlers complete
[2014-08-17T20:31:55+00:00] ERROR: Exception handlers complete
[2014-08-17T20:31:55+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 16 resources updated in 65.069223028 seconds
[2014-08-17T20:31:55+00:00] ERROR: jenkins_private_key_credentials[jenkins](jenkinsstack::master line 35) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of /usr/lib/jvm/java/bin/java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 groovy /tmp/groovy20140817-15019-10im3k2 ----
STDOUT:
STDERR: hudson.security.AccessDeniedException2: anonymous is missing the Overall/RunScripts permission
at hudson.security.ACL.checkPermission(ACL.java:55)
at hudson.model.Node.checkPermission(Node.java:417)
at hudson.cli.GroovyCommand.run(GroovyCommand.java:74)
at hudson.cli.CLICommand.main(CLICommand.java:234)
at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:309)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:290)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:249)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at hudson.cli.CliManagerImpl$1.call(CliManagerImpl.java:63)
at hudson.remoting.InterceptingExecutorService$2.call(InterceptingExecutorService.java:95)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
---- End output of /usr/lib/jvm/java/bin/java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 groovy /tmp/groovy20140817-15019-10im3k2 ----
Ran /usr/lib/jvm/java/bin/java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 groovy /tmp/groovy20140817-15019-10im3k2 returned 255
[2014-08-17T20:31:55+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

After some research I can get chef-client to succeed if I set to false in config.xml but that doesn't seem the right way to fix this. I also found this but I am not 100% sure its related:

https://github.com/opscode-cookbooks/jenkins#public-service-announcment

Initial build works fine, but if /var/lib/jenkins is copied over from another server for migration purposes and the keys inside /var/lib/jenkins/.ssh/ are overwritten, the jenkins_slave_ssh_pubkey is not updated leading to failure to converge.

Turning security off does not resolve this issue as the new key will not be updated in the node attributes.

Here is the stacktrace generated when the keys change on disk:

Mixlib::ShellOut::ShellCommandFailed: jenkins_user[chef](jenkinsstack::master line 27) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa groovy /tmp/groovy20140819-12555-1202b3w ----
STDOUT:
STDERR: Authentication failed. No private key accepted.
---- End output of java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa groovy /tmp/groovy20140819-12555-1202b3w ----
Ran java -jar /var/chef/cache/jenkins-cli.jar -s http://localhost:8080 -i /var/lib/jenkins/.ssh/id_rsa groovy /tmp/groovy20140819-12555-1202b3w returned 255
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.4.0/lib/mixlib/shellout.rb:257:in invalid!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.4.0/lib/mixlib/shellout.rb:244:inerror!'
/var/chef/cache/cookbooks/jenkins/libraries/_executor.rb:79:in execute!' /var/chef/cache/cookbooks/jenkins/libraries/_executor.rb:109:ingroovy!'
/var/chef/cache/cookbooks/jenkins/libraries/user.rb:96:in block (2 levels) in <class:JenkinsUser>' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/mixin/why_run.rb:52:incall'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/mixin/why_run.rb:52:in add_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/provider.rb:156:inconverge_by'
/var/chef/cache/cookbooks/jenkins/libraries/user.rb:95:in block in <class:JenkinsUser>' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/provider/lwrp_base.rb:138:ininstance_eval'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/provider/lwrp_base.rb:138:in block in action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/provider.rb:121:inrun_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource.rb:648:in run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/runner.rb:49:inrun_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/runner.rb:81:in block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/runner.rb:81:ineach'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/runner.rb:81:in block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection.rb:98:inblock in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection/stepable_iterator.rb:116:in call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection/stepable_iterator.rb:116:incall_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection/stepable_iterator.rb:104:initerate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/resource_collection.rb:96:inexecute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/runner.rb:80:in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/client.rb:345:inconverge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/client.rb:431:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/client.rb:213:inblock in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/client.rb:207:in fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/client.rb:207:inrun'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/application.rb:236:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/application/client.rb:338:inblock in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/application/client.rb:327:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/application/client.rb:327:inrun_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/lib/chef/application.rb:55:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.14.6/bin/chef-client:26:in<top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in

I apologize, I lost the chef-client output before creating this issue, but the problem is reproducible by simply updating the ssh keys on disk and attempting a chef-client run. It would be nice if this cookbook anticipated potential migration of /var/lib/jenkins from another server.

Perhaps use https://supermarket.getchef.com/cookbooks/nginx-proxy.

• Verify the search finds a master and public key, places key in authorized_keys
• Verify a jenkins user, group, homedir, and ~/.ssh are created

Use rbenv cookbook's LWRPs and ruby_build recipe from it, if necessary. Jenkins builds will have to source the file that activates it. We'll need this for most builds.

Verify it converges and passes tests.

• Verify jenkins user, group, homedir, and ~/.ssh are created
• Verify an SSH key is generated
• Verify plugins are installed
• (Can't verify slaves are configured, cuz Jenkins doesn't restart for some reason... but test this again with the older/current pinned Jenkins version!)

Please ignore

Get this stack added to our jenkins server.

Audit what SSH connections are allowed, only open the web interface for the reverse proxy on the master to the world.

It seems we are using a non LTS version for Jenkins :

We are currently having availability issues on the package 1.555 http://mirrors.jenkins-ci.org/war/1.555/
the LTS ones work well though.

Should we focus on stable releases only ? http://mirrors.jenkins-ci.org/war-stable/

Identify all sources of log data that can be fed to syslog or a logstash agent. Tie these back into platformstack (and whatever logging it has decided to configure) to be sure they are configured.

See rackspace-cookbooks/platformstack#134.

Verify platform converges and passes tests.