mlafeldt / chef-runner Goto Github PK

View Code? Open in Web Editor NEW

179.0 179.0 17.0 526 KB

The fastest way to run Chef cookbooks

License: Apache License 2.0

Go 70.89% Shell 25.65% Makefile 1.49% Ruby 1.63% Batchfile 0.34%

chef-runner's People

Contributors

Stargazers

Watchers

Forkers

dwabraxus mbanton harrisony landylee007 pkgr ronakbanka csharris pnaicker333 rajthilakmca neumachen gophersgang linearregression sunanxiang sacontreras psimoesssimoes blatong

chef-runner's Issues

Version Pin

Upon installing chef using curl -L https://www.opscode.com/chef/install.sh | bash I encounter the following WARNING:

WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

You are installing an omnibus package without a version pin.  If you are installing
on production servers via an automated process this is DANGEROUS and you will
be upgraded without warning on new releases, even to new major releases.
Letting the version float is only appropriate in desktop, test, development or
CI/CD environments.

WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

How do you install an omnibus package with a version pin?

When executing chef-runner second time: rsync permission errors

It tries to rsync the cookbooks but there appear a big list of delete_file permission errors.
The strange thing is that it works the first time.

rsync: delete_file: unlink(local-mode-cache/cache/cookbooks/yum/templates/default/repo.erb) failed: Permission denied (13)
rsync: delete_file: unlink(local-mode-cache/cache/cookbooks/yum/templates/default/main.erb) failed: Permission denied (13)

Improvement: Support for syncfolder

Instead of rsync all cookbooks, is it possible to use vagrant shared folder functionality?

Chef-Zero error

INFO: Preparing local files
INFO: Installing cookbook dependencies with Directory resolver
INFO: Stripping non-cookbook files
INFO: Uploading local files to machine. This may take a while...
INFO: Skipping installation of Chef
INFO: Running Chef using Vagrant driver (machine: default)
[2016-07-27T12:01:05+00:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /tmp/chef-runner
  One version per cookbook

[2016-07-27T12:01:05+00:00] INFO: Forking chef instance to converge...
Starting Chef Client, version 12.12.15
Starting Chef Client, version 12.12.15
[2016-07-27T12:01:05+00:00] INFO: *** Chef 12.12.15 ***
[2016-07-27T12:01:05+00:00] INFO: Platform: x86_64-linux
[2016-07-27T12:01:05+00:00] INFO: Chef-client pid: 5496
[2016-07-27T12:01:12+00:00] INFO: #<ChefZero::RestErrorResponse: 404: Object not found: chefzero://localhost:8889/nodes/mynode.box>
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_base.rb:91:in `rescue in get_data'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_base.rb:83:in `get_data'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/endpoints/rest_object_endpoint.rb:18:in `get'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_base.rb:62:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_router.rb:24:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/server.rb:664:in `block in app'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/server.rb:336:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/server.rb:336:in `handle_socketless_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/socketless_server_map.rb:87:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/socketless_server_map.rb:33:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http/socketless_chef_zero_client.rb:154:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:305:in `block in send_http_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:336:in `block in retrying_http_errors'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:334:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:334:in `retrying_http_errors'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:299:in `send_http_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:144:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:111:in `get'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/node.rb:585:in `load'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/node.rb:569:in `find_or_create'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/policy_builder/dynamic.rb:72:in `load_node'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/client.rb:467:in `load_node'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/client.rb:269:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:286:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:274:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:274:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:239:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:227:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:456:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:443:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:442:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:442:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:426:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:59:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/solo.rb:217:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/bin/chef-solo:25:in `<top (required)>'
/usr/bin/chef-solo:52:in `load'
/usr/bin/chef-solo:52:in `<main>'

Ansible module for chef-runner

The main goal of chef-runner is to provision remote machines via SSH. Ansible works in a similar way. So let's combine both to create the easiest Chef provisioner there is.

I already spiked an Ansible module which allows to provision a Vagrant machine. Here's a sample playbook:

---
- hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Provision system with chef-runner
      chef_runner:
        vagrant_machine: default
        install_chef: "true"
        executable: "/usr/local/bin/chef-runner"
        working_dir: "/Users/mlafeldt/devel/projects/practicing-ruby-cookbook"
        run_list:
          - recipe[apt::default]
          - practicingruby::_deploy_user

This basically works, but I lack the knowledge to complete it the way I want.

Problem 1

In particular, I don't know how to best support all the target systems chef-runner supports:

Vagrant machines
Test Kitchen instances
plain SSH
provision host system

I don't like the idea of having a separate configuration variable, e.g. vagrant_machine, for every supported system.

Problem 2

To implement the most obvious use case with Ansible -- plain SSH -- I'd somehow have to pass Ansible's SSH connection settings (hostname, user, and other SSH options) to chef-runner. For this, I need to know if it's possible to access SSH connection settings for a host from Ansible modules.

(Instead of running chef-runner locally, I could instead run it on the remote host that is going to be provisioned. Unfortunately, this would require other Ruby tools to be installed on the remote host as well, e.g. for resolving cookbook dependencies. That's why I prefer connection: local here.)

Support test-kitchen's regex for platform names

Thanks for test-kitchen support!

I'm not sure if it is technically possible, but I would appreciate it, if the syntax for specifying platform names would be in line with the one of test-kitchen.

I can do

kitchen converge debian

while I have to specify exactly

chef-runner default-debian-76

Or would this wildcard-matching contrary to chef-runners approach, because c-r always only targets one platform, not multiple (as possible with t-k)?

Feel free to directly close it, if it makes no sense to you!

Regards
Steffen

Support chef-zero

In addition to chef-solo, allow to provision systems with chef-zero.

Zero mode could be enabled like this (later it may even replace chef-solo as a default):

$ chef-runner -z

rsync trouble

$ chef-runner --rsync -v --rsync -v -H ec2-user@redacted.$cloud --ssh 'RequestTTY force' node_exporter
INFO: Starting chef-runner (v0.9.0 darwin/amd64)
INFO: Run list is [node_exporter]
INFO: Preparing local files
INFO: Installing cookbook dependencies with Directory resolver
INFO: Stripping non-cookbook files
INFO: Uploading local files to machine. This may take a while...
opening connection using ssh -l ec2-user -o "RequestTTY force" redacted.cloud rsync --server -vvlogDtprz --delete . /tmp/chef-runner
building file list ...
done
unknown message 57:6176862 [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(307) [sender=2.6.9]
ERROR: exit status 12

support arbitrary chef-solo options, or at least, minimal-ohai

Berksfile load error

My current working directory is m_java (cookbook) where Berksfile is present but runnning chef-runner is not able to load Berksfile and preventing further execution. Please assist me with necessary steps

Distribute chef-runner as .deb package

Use fpm/fpm-cookery to build .deb package
Push packages to https://packagecloud.io/mlafeldt/chef-runner

See https://github.com/mperham/inspeqtor/blob/master/Makefile

Distribute chef-runner as .rpm package

Use fpm/fpm-cookery to build .rpm package (based on #48)
Push packages to https://packagecloud.io/mlafeldt/chef-runner
Update docs

Use run_list from .kitchen.yml with -K

When using -K option, we should also load the run list defined for given instance.

I could give a go at implementing this if you agree?

Command-line option to specify resolver

At the moment, the resolver that is used to set up cookbook dependencies is automatically selected based on the files in the current directory. There are cases where more than one resolver could be used, e.g. if there's a Berksfile and a Cheffile. Provide a command-line option to specify which resolver to use. For example like this:

$ chef-runner --resolver berkshelf
$ chef-runner --resolver librarian
$ chef-runner --resolver dir

Need to be able to pass options to SSH command

I need to run Chef script on servers that sit behind a proxy. So I need to be able to pass the -o 'ProxyCommand nc -x <host>:<port> %h %p' option to the SSH command.

Something like:

> chef-runner -o 'ProxyCommand nc -x myproxy.com:7777 %h %p' -H [email protected] apt::default

Thanks!

chef-runner fails with berks 3.1.3 vendoring cookbooks

Using ChefDK and chef-runner I get the following:

$ chef-runner -M 938a943
INFO: Starting chef-runner (v0.5.0 darwin/amd64)
INFO: Preparing local files
INFO: Installing cookbook dependencies with Berkshelf resolver
DEPRECATED: `berks install --path [PATH}` has been replaced by `berks vendor`.
DEPRECATED: Re-run your command as `berks vendor [PATH]` or see `berks help vendor`.
ERROR: exit status 1

$ berks -v
3.1.3
$ chef-runner --version
chef-runner v0.5.0 darwin/amd64

Using this cookbook and a machine provisioned by test kitchen.

Using the binary distribution of chef-runner.

Let me know if there is any other debugging info that would help.

Also, it would be nice to ignore Gemfiles (since i'm using chefdk, i don't need to bundle install)

Use chef-runner as simple chef launcher

I may be totally about project and vision, so just let me know :)

But, why not using this awesome project as a simple chef runner.
For example: I want to install nodejs from latest sources.
Why not using directly chef for that?
Let's do (from anywhere): sudo chef-runner nodejs::nodejs_from_source. And hop, I have a nodejs installed.
This may be particularly useful in Makefile for some project who don't want a complete chef, but want simplify installation.

For the moment is fail with error: metadata.rb not found.

Another thing (may not be linked): I'm using Vagrant + vagrant-berkshelf for my dev.
They have a repo "devbox", with Berksfile, Berksfile.lock and Vagrantfile... and that's all!
With this, they can build a new machine for there use, they have latest (tested and fixed by .lock) version of production infra (To be speedy we are using pre-created box.. but it's not an obligation).

But in this case we have same problem: error: metadata.rb not found even if it's a good env for vagrant provision :/

Update all cookbooks if Berksfile, Cheffile, etc. have changed

If the current folder is a cookbook and its dependencies have already been resolved, only this cookbook will be updated with rsync. While this is much faster than running e.g. Berkshelf, the directory resolver should only be used if no updates to Berksfile, Cheffile, etc. have been made. At the moment, the only workaround is to delete .chef-runner.

This could be fixed by writing a timestamp file and comparing that to the timestamp of all files that are dependencies.

Use native SSH implementation

Similar to e.g. https://github.com/rapidloop/rtop-bot

cannot load such file -- berkshelf

Failing with

INFO: Starting chef-runner (v0.8.0 darwin/amd64)
INFO: Run list is [cookbooks/codio]
INFO: Preparing local files
INFO: Installing cookbook dependencies with Berkshelf resolver
/usr/local/opt/rbenv/versions/2.1.3/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- berkshelf (LoadError)
    from /usr/local/opt/rbenv/versions/2.1.3/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from -e:1:in `<main>'
ERROR: exit status 1

I have Chef and Berkshelf installed as part of ChefDK. Is ChefDK supported?

support definition of run_list in chef.json

Proposal: Is it possible to define a run_list in the chef-runner -j chef.json call other than put it in as command line args?

{
  "example": {
    "attr": "val"
  },
  "run_list": [
    "recipe[example::default]",
    "recipe[example::sub]"
  ]
}

This way we could write out this file and use it either in the repos Vagrantfile (using the traditional vagrant up/vagrant provision methods without chef-runner)

Vagrant.configure('2') do |config|
  # Configure Chef Solo provisioner
  config.vm.provision 'chef_solo' do |chef|
    # Load node attributes and run list from a JSON file
    json_file =
    if File.exist?(abspath('chef.json'))
      abspath('chef.json')
    end
    chef.json = JSON.parse(IO.read(json_file))
  end  
end

or for the chef-runner users.

chef-runner -j chef.json

Move documentation to Wiki

chef-runner's README is too long making it too hard to find relevant information in it. Move most documentation from the README to the chef-runner Wiki. Only keep most essential information in README; refer to Wiki instead.

Local mode

Allow to use chef-runner to provision the local system.

There's already been some progress: