mlafeldt / chef-runner Goto Github PK
View Code? Open in Web Editor NEWThe fastest way to run Chef cookbooks
License: Apache License 2.0
The fastest way to run Chef cookbooks
License: Apache License 2.0
Upon installing chef using curl -L https://www.opscode.com/chef/install.sh | bash
I encounter the following WARNING:
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
You are installing an omnibus package without a version pin. If you are installing
on production servers via an automated process this is DANGEROUS and you will
be upgraded without warning on new releases, even to new major releases.
Letting the version float is only appropriate in desktop, test, development or
CI/CD environments.
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
How do you install an omnibus package with a version pin?
It tries to rsync the cookbooks but there appear a big list of delete_file permission errors.
The strange thing is that it works the first time.
rsync: delete_file: unlink(local-mode-cache/cache/cookbooks/yum/templates/default/repo.erb) failed: Permission denied (13)
rsync: delete_file: unlink(local-mode-cache/cache/cookbooks/yum/templates/default/main.erb) failed: Permission denied (13)
Instead of rsync all cookbooks, is it possible to use vagrant shared folder functionality?
INFO: Preparing local files
INFO: Installing cookbook dependencies with Directory resolver
INFO: Stripping non-cookbook files
INFO: Uploading local files to machine. This may take a while...
INFO: Skipping installation of Chef
INFO: Running Chef using Vagrant driver (machine: default)
[2016-07-27T12:01:05+00:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /tmp/chef-runner
One version per cookbook
[2016-07-27T12:01:05+00:00] INFO: Forking chef instance to converge...
Starting Chef Client, version 12.12.15
Starting Chef Client, version 12.12.15
[2016-07-27T12:01:05+00:00] INFO: *** Chef 12.12.15 ***
[2016-07-27T12:01:05+00:00] INFO: Platform: x86_64-linux
[2016-07-27T12:01:05+00:00] INFO: Chef-client pid: 5496
[2016-07-27T12:01:12+00:00] INFO: #<ChefZero::RestErrorResponse: 404: Object not found: chefzero://localhost:8889/nodes/mynode.box>
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_base.rb:91:in `rescue in get_data'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_base.rb:83:in `get_data'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/endpoints/rest_object_endpoint.rb:18:in `get'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_base.rb:62:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/rest_router.rb:24:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/server.rb:664:in `block in app'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/server.rb:336:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/server.rb:336:in `handle_socketless_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/socketless_server_map.rb:87:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.7.1/lib/chef_zero/socketless_server_map.rb:33:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http/socketless_chef_zero_client.rb:154:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:305:in `block in send_http_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:336:in `block in retrying_http_errors'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:334:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:334:in `retrying_http_errors'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:299:in `send_http_request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:144:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/http.rb:111:in `get'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/node.rb:585:in `load'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/node.rb:569:in `find_or_create'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/policy_builder/dynamic.rb:72:in `load_node'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/client.rb:467:in `load_node'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/client.rb:269:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:286:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:274:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:274:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:239:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:227:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:456:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:443:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:442:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:442:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/client.rb:426:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application.rb:59:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/lib/chef/application/solo.rb:217:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.12.15/bin/chef-solo:25:in `<top (required)>'
/usr/bin/chef-solo:52:in `load'
/usr/bin/chef-solo:52:in `<main>'
The main goal of chef-runner is to provision remote machines via SSH. Ansible works in a similar way. So let's combine both to create the easiest Chef provisioner there is.
I already spiked an Ansible module which allows to provision a Vagrant machine. Here's a sample playbook:
---
- hosts: localhost
connection: local
gather_facts: no
tasks:
- name: Provision system with chef-runner
chef_runner:
vagrant_machine: default
install_chef: "true"
executable: "/usr/local/bin/chef-runner"
working_dir: "/Users/mlafeldt/devel/projects/practicing-ruby-cookbook"
run_list:
- recipe[apt::default]
- practicingruby::_deploy_user
This basically works, but I lack the knowledge to complete it the way I want.
In particular, I don't know how to best support all the target systems chef-runner supports:
I don't like the idea of having a separate configuration variable, e.g. vagrant_machine
, for every supported system.
To implement the most obvious use case with Ansible -- plain SSH -- I'd somehow have to pass Ansible's SSH connection settings (hostname, user, and other SSH options) to chef-runner. For this, I need to know if it's possible to access SSH connection settings for a host from Ansible modules.
(Instead of running chef-runner locally, I could instead run it on the remote host that is going to be provisioned. Unfortunately, this would require other Ruby tools to be installed on the remote host as well, e.g. for resolving cookbook dependencies. That's why I prefer connection: local
here.)
Thanks for test-kitchen support!
I'm not sure if it is technically possible, but I would appreciate it, if the syntax for specifying platform names would be in line with the one of test-kitchen.
I can do
kitchen converge debian
while I have to specify exactly
chef-runner default-debian-76
Or would this wildcard-matching contrary to chef-runners approach, because c-r always only targets one platform, not multiple (as possible with t-k)?
Feel free to directly close it, if it makes no sense to you!
Regards
Steffen
In addition to chef-solo, allow to provision systems with chef-zero.
Zero mode could be enabled like this (later it may even replace chef-solo as a default):
$ chef-runner -z
$ chef-runner --rsync -v --rsync -v -H ec2-user@redacted.$cloud --ssh 'RequestTTY force' node_exporter
INFO: Starting chef-runner (v0.9.0 darwin/amd64)
INFO: Run list is [node_exporter]
INFO: Preparing local files
INFO: Installing cookbook dependencies with Directory resolver
INFO: Stripping non-cookbook files
INFO: Uploading local files to machine. This may take a while...
opening connection using ssh -l ec2-user -o "RequestTTY force" redacted.cloud rsync --server -vvlogDtprz --delete . /tmp/chef-runner
building file list ...
done
unknown message 57:6176862 [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(307) [sender=2.6.9]
ERROR: exit status 12
See https://github.com/mperham/inspeqtor/blob/master/Makefile
When using -K
option, we should also load the run list defined for given instance.
I could give a go at implementing this if you agree?
At the moment, the resolver that is used to set up cookbook dependencies is automatically selected based on the files in the current directory. There are cases where more than one resolver could be used, e.g. if there's a Berksfile
and a Cheffile
. Provide a command-line option to specify which resolver to use. For example like this:
$ chef-runner --resolver berkshelf
$ chef-runner --resolver librarian
$ chef-runner --resolver dir
I need to run Chef script on servers that sit behind a proxy. So I need to be able to pass the -o 'ProxyCommand nc -x <host>:<port> %h %p'
option to the SSH command.
Something like:
> chef-runner -o 'ProxyCommand nc -x myproxy.com:7777 %h %p' -H [email protected] apt::default
Thanks!
Using ChefDK and chef-runner I get the following:
$ chef-runner -M 938a943
INFO: Starting chef-runner (v0.5.0 darwin/amd64)
INFO: Preparing local files
INFO: Installing cookbook dependencies with Berkshelf resolver
DEPRECATED: `berks install --path [PATH}` has been replaced by `berks vendor`.
DEPRECATED: Re-run your command as `berks vendor [PATH]` or see `berks help vendor`.
ERROR: exit status 1
$ berks -v
3.1.3
$ chef-runner --version
chef-runner v0.5.0 darwin/amd64
Using this cookbook and a machine provisioned by test kitchen.
Using the binary distribution of chef-runner.
Let me know if there is any other debugging info that would help.
Also, it would be nice to ignore Gemfiles (since i'm using chefdk, i don't need to bundle install)
I may be totally about project and vision, so just let me know :)
But, why not using this awesome project as a simple chef runner.
For example: I want to install nodejs from latest sources.
Why not using directly chef for that?
Let's do (from anywhere): sudo chef-runner nodejs::nodejs_from_source
. And hop, I have a nodejs installed.
This may be particularly useful in Makefile
for some project who don't want a complete chef, but want simplify installation.
For the moment is fail with error: metadata.rb not found
.
Another thing (may not be linked): I'm using Vagrant + vagrant-berkshelf for my dev.
They have a repo "devbox", with Berksfile
, Berksfile.lock
and Vagrantfile
... and that's all!
With this, they can build a new machine for there use, they have latest (tested and fixed by .lock) version of production infra (To be speedy we are using pre-created box.. but it's not an obligation).
But in this case we have same problem: error: metadata.rb not found
even if it's a good env for vagrant provision
:/
If the current folder is a cookbook and its dependencies have already been resolved, only this cookbook will be updated with rsync. While this is much faster than running e.g. Berkshelf, the directory resolver should only be used if no updates to Berksfile
, Cheffile
, etc. have been made. At the moment, the only workaround is to delete .chef-runner
.
This could be fixed by writing a timestamp file and comparing that to the timestamp of all files that are dependencies.
Similar to e.g. https://github.com/rapidloop/rtop-bot
Failing with
INFO: Starting chef-runner (v0.8.0 darwin/amd64)
INFO: Run list is [cookbooks/codio]
INFO: Preparing local files
INFO: Installing cookbook dependencies with Berkshelf resolver
/usr/local/opt/rbenv/versions/2.1.3/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- berkshelf (LoadError)
from /usr/local/opt/rbenv/versions/2.1.3/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from -e:1:in `<main>'
ERROR: exit status 1
I have Chef and Berkshelf installed as part of ChefDK. Is ChefDK supported?
Proposal: Is it possible to define a run_list in the chef-runner -j chef.json
call other than put it in as command line args?
{
"example": {
"attr": "val"
},
"run_list": [
"recipe[example::default]",
"recipe[example::sub]"
]
}
This way we could write out this file and use it either in the repos Vagrantfile (using the traditional vagrant up/vagrant provision methods without chef-runner)
Vagrant.configure('2') do |config|
# Configure Chef Solo provisioner
config.vm.provision 'chef_solo' do |chef|
# Load node attributes and run list from a JSON file
json_file =
if File.exist?(abspath('chef.json'))
abspath('chef.json')
end
chef.json = JSON.parse(IO.read(json_file))
end
end
or for the chef-runner users.
chef-runner -j chef.json
chef-runner's README is too long making it too hard to find relevant information in it. Move most documentation from the README to the chef-runner Wiki. Only keep most essential information in README; refer to Wiki instead.
Allow to use chef-runner to provision the local system.
There's already been some progress:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.