riotgamescookbooks / artifact-cookbook Goto Github PK
View Code? Open in Web Editor NEWProvides your cookbooks with the Artifact Deploy LWRP
License: Other
Provides your cookbooks with the Artifact Deploy LWRP
License: Other
I'm using the remove_top_level_directory attribute set to true to unpack the archive directly into the releases/<version>
directory. This works great but on some subsequent runs I wind up with the archive unpacked again into the releases/<version>/<artifact>
directory.
Hi,
We have a slight frustration with the fact that the restart occurs before the "current" symlink is in place. We like to refer to the app for it's service control via that symlink.
Obviously there are ways around it but I wonder if you might be able to offer the ability to have the symlink creation somewhere else?
Thanks
Screwed up the regex for:
if @new_resource.name =~ /\W/
should be \s
If the version you want is provided as latest
and the artifact_location
is not a Nexus, you will still end up with a latest
folder.
The condition in the Library should probably change to:
if version.casecmp("latest") == 0 && is_nexus?(artifact_location)
but the question is, should it?
[email protected]] out: [2013-08-07T13:42:19-04:00] FATAL: Stacktrace dumped to /tmp/chef-solo/chef-stacktrace.out
[[email protected]] out: [2013-08-07T13:42:19-04:00] FATAL: SyntaxError: /tmp/loop_install/chef/cookbooks/artifact/providers/PaxHeader/deploy.rb:1: syntax error, unexpected tIDENTIFIER, expecting $end
[[email protected]] out: 17 gid=995576803
[[email protected]] out: ^
Possibly related to http://tickets.opscode.com/browse/CHEF-4107?
Hi !
This is working :
artifact_file '/tmp/test-1.3.175.jar' do
location 'com.test:test:jar:1.3.175'
nexus_configuration nexus_conn
owner 'root'
group 'root'
end
This is not :
artifact_file 'test jar' do
path '/tmp/test-1.3.175.jar'
location 'com.test:test:jar:1.3.175'
nexus_configuration nexus_conn
owner 'root'
group 'root'
end
artifact_file
LWRP seems to totally ignore path option (path
doesn't seems to be mentionned once in the provider code).
Tell me if you want me to dive a bit into file.rb provider file, and sumbit a fix.
If the force option is specified, the current release directory is not rm -rfed. The implication is that if my tarball has files removed from it and I redeploy, I expect the files to be removed from the installed directory. Since the files are not removed, this expectation is not met.
My recommendation is to change:
action :deploy do
next unless new_resource.force or not deployed?
to
action :deploy do
delete_release_path_for(version.basename) if force
next unless deployed?
There will probably need to be some other changes, but this is the simplest approach.
When trying to download a snapshot version from nexus there seems to be a file naming problem. For example, when I try and download library 1.1-SNAPSHOT it does download it but it saves the file using the snapshot timestamp format.
/tmp/kitchen/cache/artifact_deploys/library/1.1-SNAPSHOT/library-1.1-20140210.180346-6.tar.gz
This causes artifact problems and generates this error.
Errno::ENOENT: artifact_deploy[library] (library::default line 43) had an error:
Errno::ENOENT: artifact_file[/tmp/kitchen/cache/artifact_deploys/library/1.1-SNAPSHOT/library-1.1-SNAPSHOT.tar.gz] (library::none line 582) had an error:
Errno::ENOENT: No such file or directory - /tmp/kitchen/cache/artifact_deploys/library/1.1-SNAPSHOT/library-1.1-SNAPSHOT.tar.gz
Is there a way to resolve that?
Not really sure why i get constant ArtifactNotFound exceptions in the AMS env. The artifact is there, it actually gets deployed, and some chef runs work fine; while subsequent runs fail with ArtifactNotFoundException. You can see 3 runs in mb-plugin: #461-#463, where 461 and 463 succeeded, and 462 failed.
checksum_valid calls Chef::Artifact.get_artifact_sha without passing in the ssl_verify flag
From the readme
migrate Proc.new {
execute "bundle exec rake db:migrate" do
environment { 'RAILS_ENV' => 'production' }
user "riot"
group "riot"
end
}
does not work for me, because it is executed in /srv/pvpnet
, and not in /srv/pvpnet/releases/whatever
.
Since the procs are just recipe_eval
'ed, it seems hard to change the execute block in the proc to add the cwd automatically, but then the example in the readme should be changed to
migrate Proc.new {
execute "bundle exec rake db:migrate" do
cwd release_path
environment { 'RAILS_ENV' => 'production' }
user "riot"
group "riot"
end
}
Same thing for before_migrate proc.
Doesn't seem like there is a way to do it currently.
Had this error occur when I definitely didn't have a single data bag in my Chef server. Looks like we are missing a check somewhere.
NoMethodError: artifact_deploy[foo] (foo::bar line 28) had an error: NoMethodError: undefined method `to_hash' for nil:NilClass
C:/chef/cache/cookbooks/artifact/libraries/chef_artifact_nexus.rb:15:in `remote'
C:/chef/cache/cookbooks/artifact/libraries/chef_artifact_nexus.rb:32:in `get_actual_version'
C:/chef/cache/cookbooks/artifact/providers/deploy.rb:57:in `load_current_resource'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/provider.rb:97:in `run_action'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource.rb:625:in `run_action'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/runner.rb:49:in `run_action'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/runner.rb:81:in `block (2 levels) in converge'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/runner.rb:81:in `each'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/runner.rb:81:in `block in converge'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/resource_collection.rb:96:in `execute_each_resource'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/runner.rb:80:in `converge'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/client.rb:433:in `converge'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/client.rb:500:in `do_run'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/client.rb:213:in `run'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/application.rb:208:in `run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/application/client.rb:312:in `block in run_application'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/application/client.rb:304:in `loop'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/application/client.rb:304:in `run_application'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/lib/chef/application.rb:66:in `run'
C:/opscode/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2-x86-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chef/bin/chef-client:23:in `load'
C:/opscode/chef/bin/chef-client:23:in `<main>'
Looks like we have a method defined - delete_previous_versions
- but its never used.
not sure if i'm doing something wrong or if it's related to #75
i publish a tar.gz artifact to my snapshots repo, containing my project jar and it's dependent jar (assembled with the sbt-pack plugin). when i publish a new artifact with the same snapshot version (but the project jar internally has changed), the artifact is not pulled down again but the cached version is reused.
as an aside (not sure if it has any bearing here), doing a /artifact/maven/redirect&v=LATEST does not work on my snapshots repository for some reason (but does on my public proxied group), so i set the version explicitly.
credentials data-bags : I couldn't get it to use custom naming for data-bags, I had to use a data-bag called "aws" with item name "_wildcard". I am not sure if it was intentional, if so then I would suggest documenting a little more explicitly instead of giving it as an example.
I could get the file downloaded but wasn't able to unzip in the right directory, I thought this fix for Windows cookbook could help : torresdal/windows@453ef82
But I am not sure if that helped because of (3)
Even though I used (2), unzip failed with a different error and I couldn't even manually unzip the file downloaded from s3. A little google and realized that when file is being downloaded and written to disk the "w" flag is used, when I changed it to "wb" finally I got everything to work i.e. download from s3 with the credentials and unzip the contents in a desired folder. A similar issue can be found here : http://stackoverflow.com/questions/12352230/using-aws-sdk-to-download-files-from-s3-encoding-not-right
The part of code I am talking about is on Line 152 of libraries/chef_artifact.rb file
artifact_file
downloads a file. On a subsequent chef-run, it will be downloaded again, because the default path is to use the remote_file
resource which downloads the file to a tmp dir and checks the checksum before copying, even if the file is already there.
artifact depends on nexus, and nexus depends on artifact.
Not that it's only "bad style", but it also causes other tools to break, e.g.:
http://permalink.gmane.org/gmane.comp.sysutils.chef.user/7557
Hi,
Currently each time you call one of the artifact's LWRP, it loads a bunch of deps and gems.
For one of my project, I'm working with a large number of artifacts and Chef takes a very long time (10~20 min) only to install/update chef_gem for each lwrp.
Maybe it would be interesting to provides a single recipe which would load all needed gems and dependencies, and then makes repetitives call lighter and faster ?
Hi,
Handy cookbook, I much prefer the model of using an artifact in a repo to pulling from source control.
As far as I can see, there's no way to roll back to a previous release. If I run with version set to 1.0, then again set to 2.0, then a third time setting the version to 1.0, then version 2.0 is still in place.
I'd expect it to relink to the 1.0 release and restart the app, but instead it seems to consider it as already deployed, and skips (unless force is set to true, of course).
I can hack around it for myself, but would be handy if it did the rollback to a previous version.
Thanks,
Kief
I see that you recently accepted a pull that makes the cookbook work if anonymous access is disabled, however, I can't figure out how you used anonymous access in the first place. If I omit the username
and password
fields from the data bag or set them to blank, then nexus_cli
complains about them being missing. If I set them to "anonymous"
then anonymous_enabled?
fails because the anonymous user doesn't have the privileges to check to see if it's enabled. Does anonymous access actually work?
I think it might help if the second param would default to Chef::Artifact::NexusConfiguration.from_data_bag
https://github.com/RiotGames/artifact-cookbook/blob/master/libraries/chef_artifact_nexus.rb#L9
action :deploy always calls
new_resource.updated_by_last_action(true)
This should only be called if something actually gets updated.
Maybe call this only if you need to redploy and leave it up to proc writers to call it when their proc changes something in the system
This method was renamed to data_bag_config_for
but apparently a few references were left behind.
Is there a way, to avoid this folder structure and simply deploy within the directory which I specified in the recipes ?
extract_artifact passes xzf to tar which will fail if the tarball is not gzipped. Since there are multiple ways to zip and also the ability to leave unzipped, we need to have the ability to specify the type of tarball.
tar xzf
Should check the symlink that there are files beneath it.
https://github.com/RiotGames/artifact-cookbook/blob/master/libraries/chef_artifact.rb#L276-L278
Should add the checksum attribute for the remote_file resource to check against.
I just upgraded to 1.10.0, however every time I run chef-client it fails.
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/artifact/resources/deploy.rb
================================================================================
NoMethodError
-------------
undefined method `chef_environment' for nil:NilClass
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/artifact/libraries/chef_artifact.rb:235:in `encrypted_data_bag_for'
/var/chef/cache/cookbooks/artifact/libraries/chef_artifact.rb:71:in `data_bag_config_for'
/var/chef/cache/cookbooks/artifact/libraries/chef_artifact_nexus_configuration.rb:6:in `from_data_bag'
/var/chef/cache/cookbooks/artifact/resources/deploy.rb:56:in `class_from_file'
Relevant File Content:
----------------------
/var/chef/cache/cookbooks/artifact/libraries/chef_artifact.rb:
228: # @return [Chef::Mash] the data bag item in Mash form
229: def encrypted_data_bag_for(node, data_bag)
230: @encrypted_data_bags = {} unless @encrypted_data_bags
231:
232: if encrypted_data_bags[data_bag]
233: return get_from_data_bags_cache(data_bag)
234: else
235>> data_bag_item = encrypted_data_bag_item(data_bag, node.chef_environment)
236: data_bag_item ||= encrypted_data_bag_item(data_bag, WILDCARD_DATABAG_ITEM)
237: data_bag_item ||= encrypted_data_bag_item(data_bag, "nexus")
238: data_bag_item ||= {}
239: @encrypted_data_bags[data_bag] = data_bag_item
240: return data_bag_item
241: end
242: end
243:
244: # @return [Hash]
Looking in the code (https://github.com/RiotGames/artifact-cookbook/blob/master/libraries/chef_artifact_nexus_configuration.rb#L6), nil is being passed in for the node object which then tries to call chef_environment on it.
This might be a different code-path as I am using the S3 deploy method in a Vagrant box (so with an encrypted data_bag, running against chef-server).
If this is just specific to the S3 case, perhaps I should convert the configuration to follow the same model as the new Nexus config? Going back to a commit before the chef_artifact_nexus_configuration was added fixes it for me.
If an attr with the artifact.s checksum s included, verify successful download before continuing
So far, the only issue that I've seen with the cookbook on Ubuntu 12.04 is that it doesn't specify the correct package names for S3 support:
--- a/chef/cookbooks/artifact/providers/deploy.rb
+++ b/chef/cookbooks/artifact/providers/deploy.rb
@@ -58,7 +58,7 @@ def load_current_resource
@artifact_location = [group_id, artifact_id, artifact_version, extension].join(':')
elsif Chef::Artifact.from_s3?(@new_resource.artifact_location)
unless Chef::Artifact.windows?
- %W{gcc make libxml2 libxslt libxml2-devel libxslt-devel}.each do |nokogiri_requirement|
+ %W{gcc make libxml2 libxslt1.1 libxml2-dev libxslt1-dev}.each do |nokogiri_requirement|
package nokogiri_requirement do
action :nothing
end.run_action(:install)
If a file is written into the artifact deployment directory during one of the Proc calls, it will be included in the manifest.
If that file is a configuration file that can/should be changed throughout the lifetime of that version of the artifact, it should not be overwritten because the manifest check caused a redeployment.
Perhaps, we should add an attribute where we can list files to skip during manifest checking. Or skip all the files that end up in a shared_directory
and make the documentation very clear.
The name of the resource is used in the artifact path but white-space is not removed. If the name contains spaces, for example, the shell commands will most likely fail or behave unpredictably
@artifact_root = ::File.join(@new_resource.artifact_deploy_path, @new_resource.name)
...
def extract_artifact
execute "extract_artifact" do
command "tar xzf #{cached_tar_path} -C #{release_path}"
user new_resource.owner
group new_resource.group
end
end
def copy_artifact
execute "copy artifact" do
command "cp #{cached_tar_path} #{release_path}"
user new_resource.owner
group new_resource.group
end
end
In the above code, if the name was "Deploying Files for User" then the command
being sent to the execute resource would be:
"cp /tmp/artifacts/Deploying Files for User /home/user"
The spaces will break the shell script and the chef-client will fatally error.
We might try to do a gsub on the name when putting it into artifact_root. Something like:
@artifact_root = ::File.join(@new_resource.artifact_deploy_path, @new_resource.name.gsub(/\W/, '_'))
The delete_previous_versions
method leaves some things to be desired. In particular,
delete_release_path_for
and delete_release_path_for
. It seems like these could be removed with a recursive :delete action on either the File or Directory resource.Can we add better error messaging around this? It confuses a lot of people.
Example, I want to download group/artifact/version/artifact-version.jar to /some/dir/differentFileName.jar
Currently artifact_file (and more specifically nexus_cli) only allows setting the destination directory
There are some possible edge cases with both actions in regards to the retrieve_artifact!
method.
Probably need to do some checking on whether or not we really need to download a file. There might also be a gap in the extract_artifact!
method and idempotency since its just an execute
block.
No matter what, we currently execute this code:
recipe_eval do
link new_resource.current_path do
to release_path
user new_resource.owner
group new_resource.group
end
end
I'd like to see a way (perhaps an additional attribute) to allow me to pre-seed and install content without switching the current symlink. This would also probably bypass the execution of any of the Procs, which may do any number of things.
It might make sense as an attribute:
artifact_deploy "my-artifact" do
artifact_location "somewhere"
version node[:cookbook][:artifact_version]
pre_seed node[:cookbook][:pre_seed]
end
Or an entirely new provider / action
artifact_deploy "my-artifact" do
...
action :pre_seed
end
artifact_pre_seed "my-artifact" do
...
end
artifact_deploy "my-artifact" do
...
end
Hi,
Right now we cannot select the data_bag_secret key location and it tries to load the default one. We have multiple keys and we need to be able to specify which one to use.
The link to the "Manifest Differences Flowchart" on https://github.com/RiotGames/artifact-cookbook is broken (below the main flowchart). It results in a 404 Not Found page.
Should gather some metrics on the following, because I just don't know.
When you have a large, compressed artifact...
windows_zipfile
is a pure Ruby implementation for .zip
files. Is there a better implementation that Windows has?The artifact_package resource fails to detect the extension of the file that it will be downloading. This causes the following exception:
2014-03-07T16:28:17+00:00] ERROR: artifact_package[package] (cookbook::recipe line 43) had an error: NameError: Cannot find a resource for ext on centos version 6.5
For testing purposes, it is nice to see what sequential chef runs will do when deploying the same artifact but incrementing the version number in the artifact_deploy resource. When doing so, the restart proc does not run. It seems that the restart proc will only run when the manifest is changed and not when only a symlink is changing.
SSL is important, but perhaps broadly assuming all SSL communications with Nexus should be verified was the wrong path.
In before_symlink
I have a Proc which creates some files in the shared directory if they don't exist, or copies them from our pre-Chef-based install (which had a slightly different directory layout.) This part appears to be working, as after it all falls apart you can see the files in question there in the shared directory. One such file is /opt/benbria/nodeloop-deploy/shared/.migrate-migrating
.
artifact_deploy gets to the "symlink" stage and then dies with:
[2013-08-01T12:17:20-04:00] FATAL: NoMethodError: artifact_deploy[nodeloop-deploy](nodeloop::install line 141) had an error: NoMethodError: directory[/opt/benbria/nodeloop-deploy/shared/.migrate-migrating](/tmp/loop_install/chef/cookbooks/artifact/providers/deploy.rb line 485) had an error: NoMethodError: undefined method `checksum' for Chef::Resource::Directory
Full output here:
https://gist.github.com/jwalton/da1634758061e190f049
It looks like the "directory" in question is really the one defined on line 485, but I don't see where checksum
is coming from here???
This is from artifact-cookbook 1.7.1, BTW.
If you go from:
artifact_deploy "artifact_test" do
version node[:artifact_test][:version]
artifact_location node[:artifact_test][:location]
artifact_checksum node[:artifact_test][:checksum]
deploy_to "/srv/artifact_test"
owner "artifact"
group "artifact"
symlinks({"directory_that_should_never_exist" => "directory_that_should_never_exist"})
action :deploy
end
to:
artifact_deploy "artifact_test" do
version node[:artifact_test][:version]
artifact_location node[:artifact_test][:location]
artifact_checksum node[:artifact_test][:checksum]
deploy_to "/srv/artifact_test"
owner "artifact"
group "artifact"
symlinks({"foo" => "directory_that_should_never_exist"})
action :deploy
end
The symlink won't be changed unless force is set to true.
The file provider when passed a nexus artifact location, will fail when attempting to verify the file if the nexus artifact co-ordinates contain a LATEST string for the version of the artifact.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.