lusis / chef-kibana Goto Github PK

View Code? Open in Web Editor NEW

86.0 86.0 143.0 346 KB

A Chef cookbook for Kibana3

Ruby 85.34% HTML 14.27% Shell 0.39%

chef-kibana's People

Contributors

Stargazers

Watchers

Forkers

mouadino rackspace-cookbooks marketplacer clarkdave paulczar ravwar secondmarket-cookbooks lum benwtr akqa-group webmat srenatus zsiddique novus joshuatobin mattmencel jhtran frimik uprush mdxp cotap mattchukabam polkaspots mylons greglu in-touch gansbrest agamatrix bkw jasonsutherland heffergm wearethefoos mveitas mindsignited planio-gmbh spaceapegames gondoi klamontagne bknowles cc-sysops budnik apptopia awesometown baphled mikedon pdaukintis gswallow patato-zombie neclimdul mikedillion cgravier gazzonyx cronoxy mamluka a-chernykh danburke n1koo sini raboo rickbartra91 igorshp mikesplain jketterl zubkonst batyd-turksat sysbot dissonanz bzobrist jkburges tomvachon t5unamie digitaljedi2 gleamicus denniskong tejaycar lamroger sengt awesomescot-zz phongsi alex-chef quachd x-ion-de 4thace martinb3-cookbooks onetwotrip precisiondevelopment abdel13 mingderwang lil-cain egirdhar joola phutchins lingochamp-cookbooks miketheiron rofe-mk kyle-house-sp c4tech mevansam patcon vibrato

chef-kibana's Issues

Stick to a version of kibana?

Yesterday, the Kibana master was broken for a couple of hours:
elastic/kibana#675

which means any new installs with the cookbook would be broken.

There were also a couple of changes that broke it during the last few weeks.

Do you think this cookbook could stick to a version and update from time to time, until Kibana 3 is stabilized or has a stable branch, or maybe a note could be added to the README about the instability of master?

Push releases to Community Site

Having tags here is great, and it would be even more awesome if releases were also pushed to the Community Site. I realize there's already a kibana cookbook there, but it'd make bringing this cookbook into a wrapper cookbook much more sane (i.e., using Berkshelf).

To ease this, you might look at using Stove, which can automate the versioning, changelog editing, tagging, releasing, and pushing to the Community site.

Pin back to poise ~> 1.x or make compatible with 2.x

Seeing some folks report issues with this cookbook, e.g. rackspace-cookbooks/elkstack#149. These seem to boil down to recent changes to poise.

Documentation Confusion

The readme states:

node['kibana']['install_type'] - The type of install we are going to use either git or zipfile

while, the attributes file states:

default['kibana']['install_type'] = "git" # git | file

Is it 'zipfile', or is 'file' correct?

Consider Adding to the OpsCode Chef Repo

Would you add this to the Opcode repo ? I'm doing testing in Chef-Solo, and it's a lot easier to just add a line in metadata.rb file than to upload manually.

Parameter server_name duplicated on apache.rb recipe.

https://github.com/lusis/chef-kibana/blob/a6dd02d0819d906175a553170ba0532a87be9684/recipes/apache.rb

web_app "#{node['kibana']['webserver_hostname']}-#{node['kibana']['webserver_port']}" do
  cookbook "kibana"
  docroot node['kibana']['installdir']
  server_name node['kibana']['webserver_hostname']
  template node['kibana']['apache']['template']
  es_server node['kibana']['es_server']
  es_port node['kibana']['es_port']
  server_name node['kibana']['webserver_hostname']
  server_aliases node['kibana']['webserver_aliases']
  kibana_dir node['kibana']['installdir']
  listen_address node['kibana']['webserver_listen']
  listen_port node['kibana']['webserver_port']
end

setting updated_by_last_action

https://github.com/lusis/chef-kibana/blob/master/providers/user.rb#L26

Here, and 3 other places in that file, you set new_resource.updated_by_last_action to the value of another resource's updated_by_last_action? method. I could be mistaken, but as I understand things, the embedded resource will not have run its action yet, and thus will not have set updated_by_last_action.

In any case, it would be good to move to 'use_inline_resources' where possible. I'd be happy to provide a PR if you're interested.

Issue to overwrite the templates

I'm trying to add http basic auth to the cookbook. (for the nginx webserver)
I have created my own wrapper, added the modified template (kibana-nginx.conf.erb), and set the appropriate attributes to point to my cookbook and template file.
However this is not working, my template is completely ignored...

Not sure is this is a problem on my end (I'm using OpsWorks) or a problem with the LWRPs. (it's working fine with the code before the commit ff697c1, in which they were introduced)

The only way I've made it works is if I call kibana::nginx directly from my recipe but then all the attributes from default.rb are ignored. I can add them to my own cookbook but it doesn't feel like the best solution.
I tried to modify the config.js.erb template and this one is working.

As a chef newbie, you don't want to know how many hours I've lost on this issue :/

Merge conflict in config.js.erb

https://github.com/lusis/chef-kibana/blob/master/templates/default/config.js.erb

It looks like the file was committed with a merge conflict in it.

EC2 & virtual host

The nginx virtual host needs to include the ec2.public_hostname if in EC2 in order for Kibana to be available.

User name field is used inconsistently in user provider

Opening an issue to track PR #66 and PR #64. Both have the same fix. 😄

in 3.0.0 kibana_web lwrp default behavior changed and action :create is broken

I updated to 3.0.0 and now the default action on kibana_web is :nothing, breaking my tests since nginx is never installed or configured
I manually changed my usage to action :create and this breaks the converge with the following output.

         * kibana_web[kibana] action create

           ================================================================================
           Error executing action `create` on resource 'kibana_web[kibana]'
           ================================================================================

           NoMethodError
           -------------
           No resource or method named `notifying_block' for `Chef::Provider::KibanaWeb ""'

           Cookbook Trace:
           ---------------
           /tmp/kitchen/cache/cookbooks/kibana_lwrp/libraries/web.rb:38:in `block (2 levels) in <class:KibanaWeb>'
           /tmp/kitchen/cache/cookbooks/kibana_lwrp/libraries/web.rb:37:in `block in <class:KibanaWeb>'

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cache/cookbooks/logging/recipes/kibana.rb

            66: kibana_web 'kibana' do
            67:   type        lazy { node['kibana']['webserver'] }
            68:   docroot     docroot
            69:   es_server   node['kibana']['es_server']
            70:   kibana_port node['kibana']['java_webserver_port']
            71:   template    'kibana-nginx_file.conf.erb'
            72:   action      :create
            73: end

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cache/cookbooks/logging/recipes/kibana.rb:66:in `from_file'

           kibana_web("kibana") do
             action [:create]
             retries 0
             retry_delay 2
             default_guard_interpreter :default
             declared_type :kibana_web
             cookbook_name "logging"
             recipe_name "kibana"
             type #<Chef::DelayedEvaluator:0x000000050cf940@/tmp/kitchen/cache/cookbooks/logging/recipes/kibana.rb:67>
             docroot "/opt/kibana/current/kibana"
             es_server "server-centos-67"
             kibana_port 5601
             template "kibana-nginx_file.conf.erb"
           end

test kitchen
chef 12.6.0
centos 6.7

kibana_web is notifying service[nginx] when it doesn't exist

Even with this earlier in my chef run:

Recipe: nginx::default       
  * service[nginx] action start[2014-12-24T20:47:45+00:00] INFO: Processing service[nginx] action start (nginx::default line 24)       
 (up to date)

I'm getting this when using kibana_web on v1.3.2 of the cookbook:

    ================================================================================       
    Error executing action `create` on resource 'kibana_web[kibana]'       
    ================================================================================       

    Chef::Exceptions::ResourceNotFound       
    ----------------------------------       
    resource template[/etc/nginx/sites-available/kibana] is configured to notify resource service[nginx] with action reload, but service[nginx] cannot be found in the resource collection. template[/etc/nginx/sites-available/kibana] is defined in /tmp/kitchen/cache/cookbooks/kibana/providers/web.rb:58:in `block in class_from_file'       

    Resource Declaration:       
    ---------------------       
    # In /tmp/kitchen/cache/cookbooks/elkstack/recipes/kibana.rb       

            67: kibana_web 'kibana' do
            68:   type node['kibana']['webserver']
            69:   docroot "#{node['kibana']['install_dir']}/current"
            70:   template_cookbook node['kibana']['nginx']['template_cookbook']

        71:   template node['kibana']['nginx']['template']
            72:   es_server node['kibana']['es_server']
            73:   es_port node['kibana']['es_port']
            74:   not_if { node['kibana']['webserver'].empty? }
            75: end
            76: # end replaces 'kibana::install'
            77: 
            78: # monitoring

            79: include_recipe 'elkstack::kibana_monitoring'

           Compiled Resource:
           ------------------

       # Declared in /tmp/kitchen/cache/cookbooks/elkstack/recipes/kibana.rb:67:in `from_file'

           kibana_web("kibana") do
             action :create
             retries 0
             retry_delay 2
             guard_interpreter :default
             cookbook_name "elkstack"
             recipe_name "kibana"
             type "nginx"
             docroot "/opt/kibana/current"
             template_cookbook "elkstack"
             template "kibana-nginx.conf.erb"
             es_server "127.0.0.1"
             es_port "9200"
             server_name "kibana.localhost"
             server_aliases ["kibana"]
             listen_address "0.0.0.0"
             listen_port "80"
             es_scheme "http://"

             not_if { #code block }
           end

I realize the cookbook has moved on to 1.4 and the newer kibana, but I don't see any updates to the nginx cookbook since this started happening, so I'm trying to narrow it down to a change in the kibana cookbook (I'm pretty sure it's the addition of use_inline_resources).

uninitialized constant Chef::Resource::KibanaInstall::Poise

I am getting the following error when converging on EC2: uninitialized constant Chef::Resource::KibanaInstall::Poise

Any ideas on how to fix, outside of adding require 'poise' to the top of libraries/install.rb?

Run test-kitchen with Travis + AWS

Would love to have test-kitchen integration tests reflected in the travis badge. Any objections?

So long as you're the sort of maintainer who's moderately conservative with who you give push access on the repo, happy to set it up so that it runs against master and the PR branches in the canonical repo itself

Lemme know!

(I'd also like to get it testing against debian, but that can be another PR)

no candidate version available for ruby1.9.1-full

  * package[git] action install (up to date)
  * git[/opt/kibana] action checkout
    - clone from https://github.com/rashidkpc/Kibana.git into /opt/kibana
    - checkout ref 41a12980eea9ebc751a83b8b95d378fb1b64fb12 branch v0.2.0

  * package[ruby1.9.1-full] action install
    * No version specified, and no candidate version available for ruby1.9.1-full
================================================================================
Error executing action `install` on resource 'package[ruby1.9.1-full]'
================================================================================


Chef::Exceptions::Package
-------------------------
No version specified, and no candidate version available for ruby1.9.1-full


Resource Declaration:
---------------------
# In /root/chef-solo/cookbooks-2/kibana/recipes/default.rb

 34:   package "ruby#{node['kibana']['rubyversion']}-full"
 35: else



Compiled Resource:
------------------
# Declared in /root/chef-solo/cookbooks-2/kibana/recipes/default.rb:34:in `from_file'

package("ruby1.9.1-full") do
  action :install
  retries 0
  retry_delay 2
  package_name "ruby1.9.1-full"
  cookbook_name :kibana
  recipe_name "default"
end



[2013-11-02T00:17:01+02:00] ERROR: Running exception handlers
[2013-11-02T00:17:01+02:00] ERROR: Exception handlers complete
[2013-11-02T00:17:02+02:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 4 resources updated
[2013-11-02T00:17:02+02:00] ERROR: package[ruby1.9.1-full] (kibana::default line 34) had an error: Chef::Exceptions::Package: No version specified, and no candidate version available for ruby1.9.1-full
[2013-11-02T00:17:02+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
ERROR: RuntimeError: chef-solo failed. See output above.
bash-3.2$

why does it search for wrong package name? I'm on centos it should be just ruby1.9.1 or ruby1.9.1-dev if you need headers.

Can't upgrade existing install

If the cookbook has been run for a previous version, then the cached remote_file doesn't get downloaded when you try to change to a different version.

Doing "rm /var/chef/cache/kibana_kibana.tar.gz" on the node has to be done first.

I think maybe changing references from kibana_#{kb_args[:name]}.tar.gz to kibana_#{kb_args[:version]}.tar.gz in the provider might make sense to fix this. Starting around here: https://github.com/lusis/chef-kibana/blob/master/providers/install.rb#L58

3.0.1 converge of kibana_lwrp::install fails

The end of the run fails when you include_recipe kibana_lwrp::install with the following message

       [2016-01-25T22:25:42+00:00] ERROR: resource template[/etc/nginx/sites-available/kibana] is configured to notify resource runit_service[nginx] with action reload, but runit_service[nginx] cannot be found in the resource collection. template[/etc/nginx/sites-available/kibana] is defined in /tmp/kitchen/cache/cookbooks/kibana_lwrp/libraries/web.rb:72:in `block (2 levels) in <class:KibanaWeb>'

       [2016-01-25T22:25:42+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

test kitchen
chef 12.6.0
centos 6.7

No such file or directory - logstash.json or /opt/kibana/current/dashboards/default.json

Hello,

I am having this issue when provisioning in vagrant with chef-solo.
Where that logstaash.json file is supposed to come from?

Thanks

===============================================================================
Error executing action `create` on resource 'link[/opt/kibana/current/dashboards/default.json]'
================================================================================


Errno::ENOENT
-------------
No such file or directory - logstash.json or /opt/kibana/current/dashboards/default.json


Resource Declaration:
---------------------
# In /tmp/vagrant-chef-1/chef-solo-1/cookbooks/kibana/recipes/default.rb

 59: link "#{node['kibana']['installdir']}/current/dashboards/default.json" do
 60:   to "logstash.json"
 61:   only_if { !File::symlink?("#{node['kibana']['installdir']}/current/dashboard/default.json") }
 62: end
 63:



Compiled Resource:
------------------
# Declared in /tmp/vagrant-chef-1/chef-solo-1/cookbooks/kibana/recipes/default.rb:59:in `from_file'

link("/opt/kibana/current/dashboards/default.json") do
  retry_delay 2
  target_file "/opt/kibana/current/dashboards/default.json"
  cookbook_name :kibana
  provider Chef::Provider::Link
  retries 0
  action :create
  recipe_name "default"
  link_type :symbolic
  to "logstash.json"
  only_if { #code block }
end



[2013-09-14T03:03:37+00:00] INFO: Running queued delayed notifications before re-raising exception
[2013-09-14T03:03:37+00:00] ERROR: Running exception handlers
[2013-09-14T03:03:37+00:00] ERROR: Exception handlers complete
[2013-09-14T03:03:37+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-09-14T03:03:37+00:00] FATAL: Errno::ENOENT: link[/opt/kibana/current/dashboards/default.json] (kibana::default line 59) had an error: Errno::ENOENT: No such file or directory - logstash.json or /opt/kibana/current/dashboards/default.json
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

This is the Vagrantfile

 config.vm.define :elasticsearch_debian6 do |config|

      config.vm.box = "Debian607"
      config.vm.forward_port 9090, 9090
      config.vm.forward_port 9200, 9200

      config.vm.provision :chef_solo do |chef|
          chef.cookbooks_path = ["cookbooks"]
          chef.roles_path = "roles"
          chef.data_bags_path = "data_bags"

          chef.json = {
                "java" => {
                    "install_flavor" => "oracle",
                    "oracle" => {
                        "accept_oracle_download_terms" => true
                    }
                },
                "elasticsearch"=> {
                    "cluster_name" => "logstash",
                    "bootstrap.mlockall" => false
                },
                "kibana" => {
                    "webserver" => "apache",
                    "webserver_port" => 9090
                }
          }  


          chef.add_role "elasticsearch_server"
          chef.add_role "logstash_server"
          chef.add_role "kibana_server"


      end


    end

kibana_server role just adds kibana::default recipe.

Deprecated name feature

When running the following on chef 12.5, you get the following deprecation warning.

       Deprecated features used!
         Cannot specify both default and name_property together on property path of resource yum_globalconfig. Only one (name_property) will be obeyed. In Chef 13, this will become an error. at 1 location:
           - /tmp/kitchen/cookbooks/yum/resources/globalconfig.rb:76:in `class_from_file'

Recipe to reproduce

kibana_user 'kibana' do
  name 'kibana'
  group 'kibana'
  home '/opt/kibana'
end

kibana_install 'kibana' do
  user 'kibana'
  group 'kibana'
  install_dir '/opt/kibana'
  install_type 'file'
end

kibana_install 'kibana' do
  user 'kibana'
  group 'kibana'
  install_dir '/opt/kibana-git'
  install_type 'git'
end

kibana_web 'kibana' do
  type 'nginx'
end

run init script fails to start

I find that on first provision, the run init script always fails.

However if you got to localhost:5601, the service is running.

 Recipe: nd-kibana::default
         * runit_service[kibana] action restart

           ================================================================================
           Error executing action `restart` on resource 'runit_service[kibana]'
           ================================================================================

           Mixlib::ShellOut::ShellCommandFailed
           ------------------------------------
           Expected process to exit with [0], but received '1'
           ---- Begin output of /usr/bin/sv restart /etc/service/kibana ----
           STDOUT: fail: /etc/service/kibana: unable to change to service directory: file does not exist
           STDERR:
           ---- End output of /usr/bin/sv restart /etc/service/kibana ----
           Ran /usr/bin/sv restart /etc/service/kibana returned 1

           Cookbook Trace:
           ---------------

           /tmp/kitchen/cookbooks/runit/libraries/provider_runit_service.rb:277:in `block in <class:RunitService>'

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cookbooks/nd-kibana/recipes/default.rb

            44: runit_service 'kibana' do
            45:   options(
            46:     user: "#{kibana_user}",
            47:     home: "#{node['kibana']['install_dir']}/current"
            48:   )
            49:   cookbook 'kibana_lwrp'
            50:   action :create
        51:   subscribes :restart, "template[#{kibana_config}]", :delayed
            52: end
            53:

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cookbooks/nd-kibana/recipes/default.rb:44:in `from_file'

           runit_service("kibana") do
             provider Chef::Provider::RunitService

             updated true
             supports {:restart=>true, :reload=>true, :status=>true}
             retries 0
             retry_delay 2
             default_guard_interpreter :default
             service_name "kibana"
             pattern "kibana"
             status_command "/usr/bin/sv status /etc/service"
             sv_bin "/usr/bin/sv"
             sv_dir "/etc/sv"
             service_dir "/etc/service"
             lsb_init_dir "/etc/init.d"
             options {:user=>"kibana", :home=>"/opt/kibana/current"}
             log true
             cookbook "kibana_lwrp"
         restart_on_update true
             run_template_name "kibana"
             log_template_name "kibana"
             check_script_template_name "kibana"
             finish_script_template_name "kibana"
             sv_templates true
             service_mirror # Declared in

           service("kibana") do
             provider Chef::Provider::Service::Simple
             action [:nothing]
             supports {:restart=>true, :reload=>true, :status=>true}
             retries 0
             retry_delay 2
             default_guard_interpreter :default
             service_name "kibana"
             pattern "kibana"
             start_command "/usr/bin/sv start /etc/service/kibana"

             status_command "/usr/bin/sv status /etc/service/kibana"
             restart_command "/usr/bin/sv restart /etc/service/kibana"
           end

             declared_type :runit_service
             cookbook_name :"nd-kibana"
             recipe_name "default"
           end


         * service[nginx] action reload
           - reload service service[nginx]

       Running handlers:
       [2015-11-29T01:41:20+00:00] ERROR: Running exception handlers
       Running handlers complete
       [2015-11-29T01:41:20+00:00] ERROR: Exception handlers complete
       Chef Client failed. 58 resources updated in 04 minutes 41 seconds
       [2015-11-29T01:41:20+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2015-11-29T01:41:20+00:00] ERROR: runit_service[kibana] (nd-kibana::default line 44) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
       ---- Begin output of /usr/bin/sv restart /etc/service/kibana ----
       STDOUT: fail: /etc/service/kibana: unable to change to service directory: file does not exist
       STDERR:
       ---- End output of /usr/bin/sv restart /etc/service/kibana ----
       Ran /usr/bin/sv restart /etc/service/kibana returned 1
       [2015-11-29T01:41:21+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Here is the recipe that reproduces this. The key is the "subscribes" line on the runit_service resource

install_type  = node['kibana']['install_type'] # 'file' or 'git'
kibana_user   = node['kibana']['user']
docroot       = "#{node['kibana']['install_dir']}/current/kibana"
kibana_config = "#{node['kibana']['install_dir']}/current/#{node['kibana'][install_type]['config']}"
es_server     = "#{node['kibana']['es_scheme']}#{node['kibana']['es_server']}:#{node['kibana']['es_port']}"

# Create kibana user and group
kibana_user 'kibana' do
  user "#{kibana_user}"
  group "#{kibana_user}"
  home '/opt/kibana'
  action :create
end

# Install kibana from 'file' or 'git'
kibana_install 'kibana' do
  user "#{kibana_user}"
  group "#{kibana_user}"
  install_dir '/opt/kibana'
  install_type "#{install_type}"
  action :create
end

# Create kibana.yml from template from base cookbook
template "#{kibana_config}" do
  source node['kibana']["#{install_type}"]['config_template']
  cookbook node['kibana']["#{install_type}"]['config_template_cookbook']
  mode '0644'
  user "#{kibana_user}"
  group "#{kibana_user}"
  variables(
    index: node['kibana']['config']['kibana_index'],
    port: node['kibana']['java_webserver_port'],
    elasticsearch: "#{es_server}",
    default_route: node['kibana']['config']['default_route'],
    panel_names:  node['kibana']['config']['panel_names']
  )
  action :create
end

include_recipe 'runit::default'

# Start kibana service, Only works if install_type = 'file', not 'git'
runit_service 'kibana' do
  options(
    user: "#{kibana_user}",
    home: "#{node['kibana']['install_dir']}/current"
  )
  cookbook 'kibana_lwrp'
  action :create
  subscribes :restart, "template[#{kibana_config}]", :delayed #****** <---- THIS IS THE CULPRIT ******#
end

# Create nginx proxy to forward port 5601 to 80.  Uses the template in the base cookbook
# See https://github.com/lusis/chef-kibana/tree/master/templates/default
kibana_web 'kibana' do
  type 'nginx'
  es_server "#{es_server}"
  server_name 'kitchen-kibana.localhost' #TODO change to variable
  server_aliases ["#{node['hostname']}","#{node['fqdn']}"]
  template "kibana-nginx_#{install_type}.conf.erb"
  action :create
end

And here is the kitchen file

---
driver:
  name: vagrant
  network:
  - ["forwarded_port", { guest: 5601, host: 5601 }]

provisioner:
  name: chef_solo

platforms:
  - name: ubuntu-12.04
    run_list:
      - recipe[apt]
  - name: ubuntu-14.04
    run_list:
      - recipe[apt]

suites:
  - name: default
    run_list:
      - recipe[java]
      - recipe[nd-kibana::default]
    attributes:
      java:
        install_flavor: "oracle"
        jdk_version: "8"
        oracle:
          accept_oracle_download_terms: true
      kibana:
        es_server: "swat-elasticsearchpool.example.com"

Document how to do load balancing for multiple ES servers

Would be good to support load blancing multiple ES servers in the nginx config.
Would look something like this ...

server {
  listen *:80 ;

  server_name kibana;
  access_log /var/log/nginx/kibana.access.log;

  location / {
    root /usr/share/kibana3;
    index index.html index.htm;
  }

  location ~ ^/_aliases$ {
    proxy_pass http://es_cluster;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_search$ {
    proxy_pass http://es_cluster;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_mapping$ {
    proxy_pass http://es_cluster;
    proxy_read_timeout 90;
  }

   location ~ ^/kibana-int/dashboard/.*$ {
    proxy_pass http://es_cluster;
    proxy_read_timeout 90;
  }
  location ~ ^/kibana-int/temp.*$ {
    proxy_pass http://es_cluster;
    proxy_read_timeout 90;
  }  
}

upstream es_cluster {
  server es01.domain.com:9200;
  server es02.domain.com:9200;
}

DocumentRoot takes one argument

Seems like document root is not being populated on provision.

Set default['kibana']['webserver'] = "apache"

On provision:
STDERR: Syntax error on line 6 of /etc/apache2/sites-enabled/vagrant-logging-80.conf:
DocumentRoot takes one argument, Root directory of the document tree
---- End output of /etc/init.d/apache2 start ----
Ran /etc/init.d/apache2 start returned 1

proxy location for aliases needs update

A recent change in how kibana forms its url for aliases will require a new location match for the proxy. It now passes the list of aliases it is interested in to the server. for nginx we will need a match on:

location ~ ^/.*/_aliases$ {

This issue is a result of bleskes 1/4/2014 commit:

elastic/kibana@4ad4627

Kibana 4.0.0-linux-x64 doesn't work

When accessing the url: http://localhost:5601 after the chef installation of kibana 4.0.0-linux-x64, I got this error:

Uncaught ReferenceError: ZeroClipboard is not defined (http://localhost:5601/index.js?_b=5888.89881).

and I don't know why.

Question on deploying a default dashboard

Hi,

I have successfully deployed a chef-driven stack in which Kibana helps me to browse applications logs.

Then, I built a dashboard in Kibana Web interface and exported it to my filesystem in order to save it.

What is the best practice to include in chef provisioning the task of adding this json file to Kibana configuration and making it the default dashboard ?

Do you perform this task in a dedicated recipe or is it possible using the chef-kibana cookbook ?

Thanks in advance and thank you for the awesome work.
cgravier

exception in 2.0.0-beta2: java.nio.file.AccessDeniedException: /var/cache/ldconfig

While trying to start up elasticsearch 2.0.0 beta2 I'm running into:

[2015-09-29 16:00:30,019][ERROR][org.elasticsearch.bootstrap] Exception
java.nio.file.AccessDeniedException: /var/cache/ldconfig
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:426)
    at java.nio.file.Files.newDirectoryStream(Files.java:413)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:179)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
    at java.nio.file.Files.walkFileTree(Files.java:2602)
    at java.nio.file.Files.walkFileTree(Files.java:2635)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:137)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:154)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:266)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Overriding the ["kibana"]["user"] attribute may fail convergence

Defaults work on systems that have www-data user but overriding the attribute will break if the user doesn't have a shell or doesn't exist. This is common when using a user like apache or nginx.

possible fix:

Remove the user from the various git/template commands and come in afterwards to change ownership of directories but only if the default user is not empty.

There is still an issue if the user doesn't exist at all but that may be outside this cookbook.

tag old releases

For example, it's annoying to have to use a ref of abbed9e32b0862f80346be0950e335ae2fc51577, we'd much rather use 1.1.5.

Also, having things on the community site is nice!

(ps, thanks for the awesome!)

template resource attribute doesn't exist

recipes/default.rb:82 template resource "user" does not exist. This should be "owner"

`getpwnam': can't find user for nginx (ArgumentError)

Compiled Resource:
------------------
# Declared in /root/chef-solo/cookbooks-2/kibana/recipes/default.rb:45:in `from_file'

git("/opt/kibana/master") do
  provider Chef::Provider::Git
  action [:sync]
  retries 0
  retry_delay 2
  destination "/opt/kibana/master"
  revision "master"
  remote "origin"
  cookbook_name :kibana
  recipe_name "default"
  repository "https://github.com/elasticsearch/kibana"
  user "nginx"
end

[2013-11-02T00:56:45+02:00] ERROR: Running exception handlers
[2013-11-02T00:56:45+02:00] ERROR: Exception handlers complete
[2013-11-02T00:56:45+02:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 2 resources updated
[2013-11-02T00:56:45+02:00] ERROR: git[/opt/kibana/master] (kibana::default line 45) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of git ls-remote "https://github.com/elasticsearch/kibana" master* ----
STDOUT:
STDERR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:175:in `getpwnam': can't find user for nginx (ArgumentError)
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:175:in `uid'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:101:in `set_user'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:262:in `block in fork_subprocess'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in `fork'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in `fork_subprocess'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:40:in `run_command'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run_command'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/shell_out.rb:39:in `shell_out'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/shell_out.rb:44:in `shell_out!'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/provider/git.rb:247:in `remote_resolve_reference'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/provider/git.rb:226:in `target_revision'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/provider/git.rb:67:in `block (2 levels) in define_resource_requirements'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/why_run.rb:232:in `call'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/why_run.rb:232:in `run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/why_run.rb:322:in `block in run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/why_run.rb:321:in `each'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/mixin/why_run.rb:321:in `run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/provider.rb:127:in `process_resource_requirements'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/provider.rb:104:in `run_action'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource.rb:625:in `run_action'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:49:in `run_action'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:81:in `block (2 levels) in converge'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:81:in `each'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:81:in `block in converge'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/resource_collection.rb:96:in `execute_each_resource'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/runner.rb:80:in `converge'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:433:in `converge'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:500:in `do_run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:199:in `block in run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:193:in `fork'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/client.rb:193:in `run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application.rb:208:in `run_chef_client'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application/solo.rb:221:in `block in run_application'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application/solo.rb:213:in `loop'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application/solo.rb:213:in `run_application'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/lib/chef/application.rb:66:in `run'
        from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.0/bin/chef-solo:25:in `<top (required)>'
        from /usr/bin/chef-solo:23:in `load'
        from /usr/bin/chef-solo:23:in `<main>'
---- End output of git ls-remote "https://github.com/elasticsearch/kibana" master* ----
Ran git ls-remote "https://github.com/elasticsearch/kibana" master* returned 1
[2013-11-02T00:56:45+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
ERROR: RuntimeError: chef-solo failed. See output above.
bash-3.2$

So I thought maybe nginx was installed incorrectly before or something (yeah I tried to install kibana2 last time) but it seems to be it wasn't installed:

[root@logstash-server ~]# yum remove nginx
Loaded plugins: fastestmirror
Setting up Remove Process
No Match for argument: nginx
Loading mirror speeds from cached hostfile
 * base: centos.vpsnet.lt
 * epel: epel.mirror.serveriai.lt
 * extras: centos.vpsnet.lt
 * updates: centos.vpsnet.lt
Package(s) nginx available, but not installed.
No Packages marked for removal

Allow for installing plugins

Feature request - Allow cookbook to install kibana plugins.

Most specifically, the timelion plugin

./bin/kibana plugin -i kibana/timelion
service kibana restart

https://www.elastic.co/blog/timelion-timeline

Plugins do not persist across kibana upgrades.

config.js cooked with wrong panel_names

Sometimes after cooking the config.js is cooked with panel_names squashed together in one string. I'm not touching attributes in terms of panel_names. It's like this and stays like this:

default['kibana']['config']['panel_names']   =  [ 
  'histogram', 'map', 'pie', 'table', 'filtering', 'timepicker', 'text', 'fields',
  'hits', 'dashcontrol','column', 'derivequeries', 'trends', 'bettermap', 'query', 'terms'
]

and it looks like this in config.js after the process:

panel_names: 
histogrammappietablefilteringtimepickertextfieldshitsdashcontrolcolumnderivequeriestrendsbettermapqueryterms

I'm not able to tell how exactly reproduce it. Any thoughts how to avoid this or what is the problem? Did you notice similar behaviour?

Chef version: chef solo 11.4.4-2
OS: Ubuntu 12.04 on AWS

Document default_app_id requirement

This recent change makes it required to pass default_app_id to the kibana.yml template. While the sample recipe has this parameter included, this requirement would be a nice thing to put in the CHANGELOG....or to have the template supply a default value if no parameter is passed in. ;)

Install nginx first

Nginx needs to be installed first or else the git directive referencing the nginx user will fail, as the user does not yet exist.

Support for Kibana 4

With Kibana 4 now entering RC stage, it's probably time to start thinking about updating this cookbook to handle the new infrastructure required for KB4. Opening this issue for tracking and having a discussion about how best to accomplish this support (keep support for KB 3? go all in on KB 4? create a separate cookbook?).

proxy pass

Kibana3 changed it's request a few months ago. So if your checking out kibana3 master.

The following file

https://github.com/lusis/chef-kibana/blob/master/templates/default/kibana-nginx.conf.erb

needs to have the following changed from

location ~ ^/_aliases$ {
proxy_pass <%= @es_scheme %><%= @es_server %>:<%= @es_port %>;
proxy_read_timeout 90;
}

location ~ /_aliases$ {
proxy_pass <%= @es_scheme %><%= @es_server %>:<%= @es_port %>;
proxy_read_timeout 90;
}

Deprecated provides declaration

       Recipe: nd-kibana::default
         * kibana_user[kibana] action create[2015-11-28T16:32:37+00:00] WARN: Class Chef::Provider::KibanaUser does not declare 'provides :kibana_user'.
       [2015-11-28T16:32:37+00:00] WARN: This will no longer work in Chef 13: you must use 'provides' to use the resource's DSL.

           * user[kibana] action create (up to date)
           * group[kibana] action create (up to date)
           - create resource kibana
         * kibana_install[kibana] action create[2015-11-28T16:32:37+00:00] WARN: Class Chef::Provider::KibanaInstall does not declare 'provides :kibana_install'.
       [2015-11-28T16:32:37+00:00] WARN: This will no longer work in Chef 13: you must use 'provides' to use the resource's DSL.

The lwrp's need to be updated to work with chef 13

Failed install kibana 3 using kibana_install

Here is the stacktrace

================================================================================
Error executing action `extract` on resource 'libarchive_file[kibana_3.1.2_web.tar.gz]'
================================================================================


LoadError
---------
cannot load such file -- ffi


Cookbook Trace:
---------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/libarchive/libraries/helper.rb:31:in `require'
/var/lib/aws/opsworks/cache.stage2/cookbooks/libarchive/libraries/helper.rb:31:in `extract'
/var/lib/aws/opsworks/cache.stage2/cookbooks/libarchive/providers/file.rb:18:in `block in class_from_file'


Resource Declaration:
---------------------
# In /var/lib/aws/opsworks/cache.stage2/cookbooks/kibana/providers/install.rb

67:       res = libarchive_file "kibana_#{kb_args[:file_version]}_#{kb_args[:name]}.tar.gz" do
68:         path "#{Chef::Config[:file_cache_path]}/kibana_#{kb_args[:file_version]}_#{kb_args[:name]}.tar.gz"
69:         extract_to kb_args[:install_dir]
70:         owner kb_args[:user]
71:         action [:extract]
72:       end
73:       new_resource.updated_by_last_action(res.updated_by_last_action?)



Compiled Resource:
------------------
# Declared in /var/lib/aws/opsworks/cache.stage2/cookbooks/kibana/providers/install.rb:67:in `block in class_from_file'

libarchive_file("kibana_3.1.2_web.tar.gz") do
action [:extract]
retries 0
retry_delay 2
cookbook_name "my_cookbook"
path "/var/lib/aws/opsworks/cache.stage2/kibana_3.1.2_web.tar.gz"
extract_to "/opt/kibana"
owner "www-data"
mode 493
end

Tag releases for easier versioning via Berkshelf

None of your versions are tagged here in the repo, which makes it a bit tricky to use them. It'd be immensely helpful if you tagged your releases.

Release new version

Cookbook is currently broken for anyone who wants to override node.kibana.es_server. The fix is in b121b67. Can we get a release please?

Github page title outdated

"A Chef cookbook for Kibana3"

"A Chef cookbook for Kibana4"

When I put dependency on kibana_lwrp cookbook in my metadata.rb I get following error

Recipe Compile Error in /var/chef/cache/cookbooks/kibana_lwrp/libraries/install.rb

LoadError

cannot load such file -- poise

Cookbook Trace:

/var/chef/cache/cookbooks/kibana_lwrp/libraries/install.rb:10:in `<top (required)>'

Relevant File Content:

/var/chef/cache/cookbooks/kibana_lwrp/libraries/install.rb:

3: # Resource/Provider:: install
4: # Author:: John E. Vincent
5: # Author:: Paul Czarkowski
6: # License:: Apache 2.0
7: #
8: # Copyright 2014, John E. Vincent
9:
10>> require 'poise'
11:
12: class Chef
13: class Resource::KibanaInstall < Resource
14: include Poise
15:
16: actions(:create, :remove)
17:
18: attribute(:name, kind_of: String, name_attribute: true)
19: attribute(:user, kind_of: String, default: 'kibana')

How do I include Poise in my chef setup? I tried to upload the cookbook but it complains that it does not have metadata.rb in the master branch and kibana_lwrp does not specify a version I should stick to. Any help on this?

Chef- installation is failing since release of new 4.0.0 beta at the elasticsearch/kibana repository

The installation is taking by default the master branch but now the directory structure was changes and I get the following error:

Error executing action create on resource 'link[/opt/kibana/current/src/app/dashboards/default.json]'
Errno::ENOENT
No such file or directory - (logstash.json, /opt/kibana/current/src/app/dashboards/default.json)

It seems that the folder stracture changed and the default.json file is missing now.

cannot disable webserver configuration

Setting node['kibana']['webserver'] to '' raises the following exception:

Chef::Exceptions::ValidationFailed
----------------------------------
Option type must be equal to one of: apache, nginx!  You passed "".


Cookbook Trace:
---------------
kibana/recipes/install.rb:65:in `block in from_file'
kibana/recipes/install.rb:64:in `from_file'`from_file'

Relevant File Content:
----------------------
kibana/recipes/install.rb:

58: 
59:  link "#{node['kibana']['install_dir']}/current/app/dashboards/default.json" do
60:    to 'logstash.json'
61:    only_if { !File.symlink?("#{node['kibana']['install_dir']}/current/app/dashboards/default.json") }
62:  end
63: 
64:  kibana_web 'kibana' do
65>>   type node['kibana']['webserver']
66:    docroot "#{node['kibana']['install_dir']}/current"
67:    es_server node['kibana']['es_server']
68:    not_if { node['kibana']['webserver'].empty? }
69:  end
70:

merge 1.2.0 branch into master

1.2.0 has a bunch of new features ... merge it into master.

Push 1.3.x cookbook versions to supermarket

Was going to submit a PR to make the installation instructions for kibana v4 vs v3 more clear, but realized that this fails:

# Berksfile

cookbook 'kibana_lwrp', '~> 1.3.0'

Any reason these couldn't be pushed? I can imagine some wonkiness on supermarket that makes pushing old versions unfavoured, but if so, then I can poked around about that.

Cheers!

EDIT: I do realize it can be done via git/github keys, but then we lose version constraints, so it's more brittle :)

cookbook wont run in AWS OPSWORKS chef 11.10

Hi
When im trying to run that cookbook on my Opsworks stack using the latest available chef version 11.10 - poise makes problems 👍 Halite is not compatible with no_lazy_load false, please set no_lazy_load true in your Chef configuration file
as poise is not nessesary for kibana per se and Opsworks still do not support chef 12 for linux stacks
can a backward compatibility be added ?

CentOS 7 fails on semanage

Please ignore.

Download from url requires 2 attributes

When downloading a version of kibana other than 4.0.0, you need to set 2 attributes

node['kibana']['version'] = "4.1.3-linux-x64"
node['kibana']['file']['url'] =  "https://download.elasticsearch.org/kibana/kibana/kibana-4.1.3-linux-x64.tar.gz"

This is counter intuitive and should be documented.

As a side note, the file that it downloads has a wonky filename

kibana_kibana_4.1.3-linux-x64_kibana.tar.gz

You can see the line that generates this filename here:

https://github.com/lusis/chef-kibana/blob/master/libraries/install.rb#L78