voxpupuli / puppet-ghostbuster Goto Github PK

View Code? Open in Web Editor NEW

91.0 13.0 25.0 182 KB

👻 Dead code detector for Puppet

License: Apache License 2.0

Ruby 100.00%

puppet code-quality cleanup puppetdb puppet-lint hacktoberfest

puppet-ghostbuster's Introduction

puppet-ghostbuster

When you have dead puppet code hanging around ... Who you gonna call ?

This gems helps puppet users to find dead code by displaying unused classes, defined resources, template and files.

This gems only support PuppetDB APi v4 (PuppetDB 3+)

Usage

$ find . -type f -exec puppet-lint --only-checks ghostbuster_classes,ghostbuster_defines,ghostbuster_facts,ghostbuster_files,ghostbuster_functions,ghostbuster_hiera_files,ghostbuster_templates,ghostbuster_types {} \+

Environment variables

HIERA_YAML_PATH

The location of the hiera.yaml file. Defaults to ./hiera.yaml or /etc/puppetlabs/puppet/hiera.yaml whichever is found first.

PUPPETDB_URL

The url or the PuppetDB. Defaults to http://puppetdb:8080

PUPPETDB_CACERT_FILE

Your site’s CA certificate

PUPPETDB_CERT_FILE

An SSL certificate signed by your site’s Puppet CA

PUPPETDB_KEY_FILE

The private key for that certificate

PE_TOKEN

If set, PE token authentication will be used instead of certificate authentication. Value may be a token or path to a file containing a token.

Plugins

ghostbuster_classes

Find unused classes in PuppetDB.

ghostbuster_defines

Find unused defined types in PuppetDB.

ghostbuster_facts

Find unused facts in Puppet manifests and templates.

ghostbuster_files

Find unused files in PuppetDB or in Puppet manifests.

ghostbuster_functions

Find unused functions in Puppet manifests or templates.

ghostbuster_hiera_files

Find unused hiera files in PuppetDB.

ghostbuster_templates

Find unused templates in Puppet manifests.

ghostbuster_types

Find unused types in Puppet manifests.

Example output

TODO

$ find . -type f -exec puppet-lint --only-checks ghostbuster_classes,ghostbuster_defines,ghostbuster_files,ghostbuster_hiera_files,ghostbuster_templates {} \+
./modules/foo/manifests/install.pp - WARNING: Class Foo::Install seems unused on line 1
./modules/foo/manifests/service.pp - WARNING: Class Foo::Service seems unused on line 1
./modules/foo/manifests/init.pp - WARNING: Class Foo seems unused on line 1
./modules/bar/manifests/baz.pp - WARNING: Define Bar::Baz seems unused on line 1
./modules/modulename/templates/foo.erb - WARNING: Template modulename/foo.erb seems unused on line 1
./modules/modulename/templates/bar.erb - WARNING: Template modulename/bar.erb seems unused on line 1
./modules/modulename/templates/baz.erb - WARNING: Template modulename/baz.erb seems unused on line 1
./modules/foo/files/bar.txt - WARNING: File foo/bar.txt seems unused on line 1
./modules/foo/files/baz.txt - WARNING: File foo/baz.txt seems unused on line 1

Transfer Notice

This plugin was originally authored by Camptocamp. The maintainer preferred that Puppet Community take ownership of the module for future improvement and maintenance. Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Camptocamp.

Previously: https://github.com/camptocamp/puppet-ghostbuster

puppet-ghostbuster's People

Contributors

Stargazers

Watchers

puppet-ghostbuster's Issues

refactor puppet db queries (self.client)

We currently make a new connection for each db query 🐌

I'm quite sure we could speed up things by reusing existent connection.
We should see how to do this with the puppetdb class.

Plugin to detect unused facts

PuppetDB 3+ support

With PuppetDB 3 and later the v2/v3 API endpoints have been retired. The puppetdb-ruby client this depends on does not support the v4 API without some minor patching but the project seems dead.

Essentially the patch I did locally was:

    PuppetDB::Client.new({
      :server => configuration.puppetdbserverurl,
      :pem    => {
        'key'     => configuration.hostprivkey,
        'cert'    => configuration.hostcert,
        'ca_file' => configuration.localcacert,
      }
    }, 4)

(notice the , 4 thing before the closing ))

And then calling it with -s https://my-puppetdb-host:8081/pdb/query appears to do the trick.

Module path with ending slash cause an error

If the _module path ends_ with a slash you get an error.

✔ 12:51 ~/workspace/puppet-ghostbuster [ master | ✚ 1 …3 ] $ puppet-ghostbuster --key ssl/private.pem --cert ssl/public.pem --ca ssl/ca.pem -s http://localhost:8080 ~/workspace/puppet/modules/module_name_with_ending_slash/
I, [2016-01-04T12:51:29.207418 #17394]  INFO -- PuppetGhostbuster: Now trying to find unused classes
I, [2016-01-04T12:51:29.224780 #17394]  INFO -- PuppetGhostbuster: Now trying to find unused defines
I, [2016-01-04T12:51:29.225225 #17394]  INFO -- PuppetGhostbuster: Now trying to find unused templates
/home/tuxinaut/workspace/puppet/modules/module_name_with_ending_slash//templates/env.erb
/home/tuxinaut/workspace/puppet-ghostbuster/lib/puppet-ghostbuster.rb:127:in `block in find_unused_templates': undefined method `captures' for nil:NilClass (NoMethodError)
    from /home/tuxinaut/workspace/puppet-ghostbuster/lib/puppet-ghostbuster.rb:124:in `each'
    from /home/tuxinaut/workspace/puppet-ghostbuster/lib/puppet-ghostbuster.rb:124:in `find_unused_templates'
    from /home/tuxinaut/workspace/puppet-ghostbuster/lib/puppet-ghostbuster.rb:180:in `run'
    from /home/tuxinaut/workspace/puppet-ghostbuster/lib/puppet-ghostbuster/bin.rb:26:in `run'
    from /home/tuxinaut/workspace/puppet-ghostbuster/bin/puppet-ghostbuster:7:in `<top (required)>'
    from /home/tuxinaut/.rvm/gems/ruby-2.2-head@puppet_ghostbuster/bin/puppet-ghostbuster:23:in `load'
    from /home/tuxinaut/.rvm/gems/ruby-2.2-head@puppet_ghostbuster/bin/puppet-ghostbuster:23:in `<main>'
    from /home/tuxinaut/.rvm/gems/ruby-2.2-head@puppet_ghostbuster/bin/ruby_executable_hooks:15:in `eval'
    from /home/tuxinaut/.rvm/gems/ruby-2.2-head@puppet_ghostbuster/bin/ruby_executable_hooks:15:in `<main>'
✔ 12:51 ~/workspace/puppet-ghostbuster [ master | ✚ 1 …3 ] $

Use fixtures for puppetdb calls

It would be cool to have a single resources.json and have a lib that mocks the calls to the PuppetDB and returns only the subset of resources matching the query.

Fix warning: class variable acce ss from toplevel

Add preconnect parameter

So that we can open an ssh tunnel to the host that hosts the puppetdb

Use vcr for unit tests

Use net/http instead of puppetdb-ruby

The puppetdb-ruby gem provides a useful client but seems unable to encode more complex queries correctly causing APIError's to be thrown. Especially the classes query could benefit from a more efficient query on larger installations.

I'm having some trouble with Ruby right now but this should more or less be it:

require 'json'
require 'net/http'
require 'ghostbusters/version'


class PuppetGhostbuster
  class PuppetDB
    def self.client
      uri = URI.parse("#{ENV['PUPPETDB_URL'] || 'http://puppetdb:8080'}")

      @@client ||= Net::HTTP.new(uri.host, uri.port)
      @@client.key = ENV['PUPPETDB_KEY_FILE']
      @@client.cert = ENV['PUPPETDB_CERT_FILE']
      @@client.ca_file = ENV['PUPPETDB_CACERT_FILE']
    end

[..]

    def self.request(endpoint, query = nil)
      uri_prefix = '/pdb/query/v4'
      headers = {'user-agent' => "ghostbusters/#{PuppetGhostbuster::VERSION}", 'accept' => 'application/json' }
      if query.nil?
        request = Net::HTTP::Get.new("#{uri_prefix}/#{endpoint}", headers)
      else
        request = Net::HTTP::Get.new("#{uri_prefix}/#{endpoint}?query=#{URI.encode(query)}", headers)
      end
      resp = client.request(request)
      case resp
      when Net::HTTPSuccess
        JSON.parse resp.body
      else
        resp.value # calling value will raise an exception if we didn't get a 200
      end
    end
[..]

   def self.classes
      query = '["extract", [["function", "count"], "title"], ["=", "type", "Class"], ["group_by", "title"]]'
      @@classes ||= client.request('resources', query).select { |r| r.title }
  end

Add support for PuppetDB 2.x

We're still on PuppetDB 2.x so it would be useful to add an option to switch between the versions of PuppetDB.

raise "ensure => absent" resources

Would be great to find the ensure => absent resources in order to be able to clean a bit the catalogs.

Skip symlink files

Refactor puppet-lint plugins

Plugin to detect unused functions

Plugin to detect unused entries in hiera files

Getting 'ArgumentError: invalid byte sequence in UTF-8' when using ghostbuster_facts

Hello

I came across the following error when using ghostbuster_facts:

PUPPETDB_URL='http://localhost:8080/' HIERA_YAML_PATH='/tmp/hiera.yaml' find . -type f -
exec puppet-lint --only-checks ghostbuster_facts {} \+
Whoops! It looks like puppet-lint has encountered an error that it doesn't
know how to handle. Please open an issue at https://github.com/rodjek/puppet-lint
and paste the following output into the issue description.
---
puppet-lint version: 2.3.3
ruby version: 2.1.9-p490
platform: x86_64-darwin16.0
file path: ./modules/apt/lib/facter/apt_updates.rb
file contents:

apt_package_updates = nil
Facter.add("apt_has_updates") do
  confine :osfamily => 'Debian'
  if File.executable?("/usr/bin/apt-get")
    apt_get_result = Facter::Util::Resolution.exec('/usr/bin/apt-get -s upgrade 2>&1')
    if not apt_get_result.nil?
      apt_package_updates = [[], []]
      apt_get_result.each_line do |line|
        if line =~ /^Inst\s/
          package = line.gsub(/^Inst\s([^\s]+)\s.*/, '\1').strip
          apt_package_updates[0].push(package)
          security_matches = [
            / Debian[^\s]+-updates /,
            / Debian-Security:/,
            / Ubuntu[^\s]+-security /,
            / gNewSense[^\s]+-security /
          ]
          re = Regexp.union(security_matches)
          if line.match(re)
            apt_package_updates[1].push(package)
          end
        end
      end
    end
  end

  setcode do
    if not apt_package_updates.nil? and apt_package_updates.length == 2
      apt_package_updates != [[], []]
    end
  end
end

Facter.add("apt_package_updates") do
  confine :apt_has_updates => true
  setcode do
    if Facter.version < '2.0.0'
      apt_package_updates[0].join(',')
    else
      apt_package_updates[0]
    end
  end
end

Facter.add("apt_updates") do
  confine :apt_has_updates => true
  setcode do
    Integer(apt_package_updates[0].length)
  end
end

Facter.add("apt_security_updates") do
  confine :apt_has_updates => true
  setcode do
    Integer(apt_package_updates[1].length)
  end
end

error:

ArgumentError: invalid byte sequence in UTF-8
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:40:in `==='
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:40:in `each'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:40:in `grep'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:40:in `block (2 levels) in check'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:39:in `each'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:39:in `block in check'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:28:in `each'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-ghostbuster-0.9.0/lib/puppet-lint/plugins/check_ghostbuster_facts.rb:28:in `check'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/checkplugin.rb:21:in `run'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/checks.rb:60:in `block in run'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/checks.rb:57:in `each'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/checks.rb:57:in `run'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint.rb:196:in `run'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/bin.rb:59:in `block in run'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/bin.rb:56:in `each'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/lib/puppet-lint/bin.rb:56:in `run'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/gems/puppet-lint-2.3.3/bin/puppet-lint:7:in `<top (required)>'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/bin/puppet-lint:23:in `load'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/bin/puppet-lint:23:in `<main>'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/bin/ruby_executable_hooks:15:in `eval'
/Users/schowdhury/.rvm/gems/ruby-2.1.9/bin/ruby_executable_hooks:15:in `<main>'

Here are the gem's I currently have loaded:

gem list

*** LOCAL GEMS ***

bigdecimal (default: 1.2.4)
bundler-unload (1.0.2)
CFPropertyList (2.3.5)
executable-hooks (1.3.2)
facter (2.5.1 universal-darwin)
fast_gettext (1.1.0)
gem-wrappers (1.3.2)
gettext (3.2.4)
gettext-setup (0.28)
hiera (3.4.2)
httparty (0.15.6)
io-console (default: 0.4.3)
json (default: 1.8.1)
locale (2.1.2)
minitest (default: 4.7.5)
multi_xml (0.6.0)
psych (default: 2.0.5)
puppet (5.3.2 universal-darwin)
puppet-ghostbuster (0.9.0)
puppet-lint (2.3.3)
puppetdb-ruby (1.1.1)
rake (default: 10.1.0)
rdoc (default: 4.1.0)
rubygems-bundler (1.4.4)
rvm (1.11.3.9)
test-unit (default: 2.1.9.0)
text (1.3.1)

Has anyone else came across this issue as well?

Add support for non SSL

non-ssl fails

Great to see this module :)

I'd love a flag to use a non-ssl integration with PuppetDB. I'm not sure I'll have any time soon to send a PR for this, but would love to see it!

Transfer to Vox Pupuli?

Hi Camp2Camp,

Would there be any interest in transferring this repository to Vox Pupuli?

With @raphink moving on, the community might worry that the project will descend further into obsoleteness. Vox Pupuli has expressed interest, and requested that I lodge this issue to determine if you would be willing to migrate?

Output to json

Plugin to detect unused top scope variables

Fix manifests filter regex

Can we bump this module to support puppet-lint > 3?

Bundler could not find compatible versions for gem "puppet-lint":
  In Gemfile:
    puppet-lint (= 3.0.1)
    puppet-ghostbuster (~> 0.9) was resolved to 0.9.0, which depends on
      puppet-lint (< 3.0, >= 1.0)

$ find -type f -exec puppet-lint --only-checks ghostbuster_classes,ghostbuster_defines,ghostbuster_facts,ghostbuster_files,ghostbuster_functions,ghostbuster_hiera_files,ghostbuster_templates,ghostbuster_types {} \+
/Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:2112:in `raise_if_conflicts': Unable to activate puppet-ghostbuster-0.7.3, because puppet-lint-2.0.2 conflicts with puppet-lint (~> 1.0) (Gem::ConflictError)
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1280:in `activate'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems.rb:198:in `rescue in try_activate'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems.rb:195:in `try_activate'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:126:in `rescue in require'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:39:in `require'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-ghostbuster-0.7.3/lib/puppet-lint/plugins/check_ghostbuster_classes.rb:1:in `<top (required)>'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:18:in `load'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:18:in `block (2 levels) in load_from_gems'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:17:in `each'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:17:in `block in load_from_gems'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:16:in `each'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:16:in `load_from_gems'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint/plugins.rb:74:in `<top (required)>'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/lib/puppet-lint.rb:214:in `<top (required)>'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/amos.shapira/.rvm/rubies/ruby-2.1.3/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/gems/puppet-lint-2.0.2/bin/puppet-lint:5:in `<top (required)>'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/bin/puppet-lint:23:in `load'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/bin/puppet-lint:23:in `<main>'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/bin/ruby_executable_hooks:15:in `eval'
    from /Users/amos.shapira/.rvm/gems/ruby-2.1.3/bin/ruby_executable_hooks:15:in `<main>'

any chance of making it work with puppet-lint 2?

Detect templates included in templates

<%= scope.function_template(['foo/bar/baz.erb']) -%>