voxpupuli / puppet-splunk Goto Github PK
View Code? Open in Web Editor NEWManage Splunk servers and forwarders using Puppet
Home Page: https://forge.puppet.com/puppet/splunk
License: Apache License 2.0
Manage Splunk servers and forwarders using Puppet
Home Page: https://forge.puppet.com/puppet/splunk
License: Apache License 2.0
class { '::splunk::params':
server => 'localhost',
}
include ::splunk::forwarder
splunkforwarder_input { "test-index":
section => "monitor://${title}",
setting => 'index',
value => 'test',
}
splunkforwarder_input { "test-sourcetype":
section => "monitor://${title}",
setting => 'sourcetype',
value => 'test',
}
Splunk gets started by puppet before the splunk config is finished. Exec['enable_splunkforwarder']
starts the splunk daemon, this might be before puppet is done writing all the config snippets. The service in splunk::virtual
a bit later fails silently because splunkd is already running. It logs errors, but systemd/puppet (and in most cases, the end user 😀) are happy since the service is "running" already.
In the logs it can be seen that the test-index
and test-sourcetype
resources are applied after license_splunkforwarder
.
Splunk running with the full config as specified through puppet.
Nov 24 08:53:33 localhost puppet-user[2616]: Compiled catalog for localhost.localdomain in environment production in 0.66 seconds
Nov 24 08:53:33 localhost puppet-user[2616]: (/Stage[main]/Archive::Staging/File[/opt/staging]/ensure) created
Nov 24 08:53:36 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/Archive[/opt/staging/splunk/splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm]/ensure) download archive from https://download.splunk.com/products/universalforwarder/releases/7.0.0/linux/splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm to /opt/staging/splunk/splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm with cleanup
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/Package[splunkforwarder]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_input[default_host]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_output[tcpout_defaultgroup]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_output[defaultgroup_server]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/Splunkforwarder_web[forwarder_splunkd_port]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/deploymentclient.conf]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/inputs.conf]/mode) mode changed '0644' to '0600'
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/inputs.conf]/seluser) seluser changed 'unconfined_u' to 'system_u'
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/outputs.conf]/mode) mode changed '0644' to '0600'
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/outputs.conf]/seluser) seluser changed 'unconfined_u' to 'system_u'
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/web.conf]/mode) mode changed '0644' to '0600'
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/web.conf]/seluser) seluser changed 'unconfined_u' to 'system_u'
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/limits.conf]/ensure) created
Nov 24 08:53:37 localhost puppet-user[2616]: (/Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/server.conf]/ensure) created
Nov 24 08:53:38 localhost puppet-user[2616]: (/Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns) executed successfully
Nov 24 08:53:38 localhost systemd: Reloading.
Nov 24 08:53:38 localhost systemd: Reloading.
Nov 24 08:53:38 localhost puppet-user[2616]: (/Stage[main]/Splunk::Platform::Posix/Exec[enable_splunkforwarder]/returns) executed successfully
Nov 24 08:53:38 localhost puppet-user[2616]: (/Stage[main]/Main/Splunkforwarder_input[test-index]/ensure) created
Nov 24 08:53:38 localhost puppet-user[2616]: (/Stage[main]/Main/Splunkforwarder_input[test-sourcetype]/ensure) created
Nov 24 08:53:38 localhost systemd: Reloading.
Nov 24 08:53:38 localhost systemd: Starting SYSV: Splunk indexer service...
Nov 24 08:53:38 localhost splunk: Starting Splunk...
Nov 24 08:53:39 localhost splunk: The splunk daemon (splunkd) is already running. [FAILED]
Nov 24 08:53:39 localhost systemd: Started SYSV: Splunk indexer service.
Nov 24 08:53:39 localhost puppet-user[2616]: (/Stage[main]/Splunk::Virtual/Service[splunk]/ensure) ensure changed 'stopped' to 'running'
Nov 24 08:53:39 localhost puppet-user[2616]: Applied catalog in 6.01 seconds
There might be several ways to fix this:
splunk stop
after starting to accept the license, this can take a while (even on a fresh and mostly empty splunk install this can exceed the timeout for the puppet service provider)Nov 24 09:14:29 localhost puppet-user[2705]: (/Stage[main]/Main/Splunkforwarder_input[test-index]/ensure) created
Nov 24 09:14:29 localhost puppet-user[2705]: (/Stage[main]/Main/Splunkforwarder_input[test-sourcetype]/ensure) created
Nov 24 09:14:32 localhost puppet-user[2705]: (/Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns) executed successfully
Nov 24 09:14:32 localhost systemd: Reloading.
Nov 24 09:14:32 localhost systemd: Reloading.
Nov 24 09:14:32 localhost puppet-user[2705]: (/Stage[main]/Splunk::Platform::Posix/Exec[enable_splunkforwarder]/returns) executed successfully
Nov 24 09:14:32 localhost systemd: Reloading.
Nov 24 09:14:32 localhost systemd: Starting SYSV: Splunk indexer service...
Nov 24 09:14:32 localhost splunk: Starting Splunk...
Nov 24 09:14:33 localhost splunk: The splunk daemon (splunkd) is already running. [FAILED]
Starting splunk for the second time still fails, but at least the config was completed before enable_splunkforwarder was run.
class {'::splunk::forwarder':
pkg_provider => 'yum',
}
Puppet gives the following warning:
Warning: Unknown variable: 'staged_package'. at /etc/puppetlabs/code/environments/production/modules/splunk/manifests/forwarder.pp:93:28
No warning.
Lets get the staging module, which is deprecated, replaced with the archive module like was done with the rabbitmq module!
I am getting the following error on splunkd.logs
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/cpu.sh" /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/cpu.sh: line 16: /common.sh: No such file or directory
Hi SETeam,
Thanks for creating this very useful puppet module.
I'm trying to modify the module so that it will upgrade universal forwarders to the latest version and can't get it to work right, so I was hoping that I could get help with that or get that functionality added to the module.
Thanks,
Alex
It would be awesome if we could implement a feature in which puppet can auto manage deployment of splunk licenses for users.
http://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/HowSplunklicensingworks
splunk_props { 'useless title':
path => '/opt/splunk/etc/master-apps/_cluster/local', # <-- would like to be able to do this
section => 'default',
setting => 'defaultGroup',
value => 'splunk_9777',
}
The resource types have hardcoded paths to the .conf
files and do not allow for specifying a path.
Allowing the path to be specified on a per-resource basis would allow for management of index and search head clusters.
Specifically, I would like to be able to manage the index cluster config files located at /opt/splunk/etc/master-apps/_cluster/local
on the cluster master.
Is this something that this module could support? If so, what would be the recommended way to implement this?
I've gotten this working by declaring additional resource types (splunkmaster_indexes
and splunkmaster_inputs
) but it feels forced. I have not been successful in implementing a path
property on the base type since it seems to be a limitation of puppetlabs-inifile
.
class { '::splunk::params':
server => 'splunkmaster.domain.name'
}
class { '::splunk::forwarder':
pkg_provider => 'yum',
package_name => 'splunkforwarder',
package_ensure => installed
}
Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/splunk_config: Could not autoload puppet/provider/splunk_distsearch/ini_setting: undefined method `provider' for nil:NilClass at /etc/puppetlabs/code/environments/splunk_client/modules/splunk/manifests/params.pp:244:3 on node i-26932f02
Installation of Splunk forwarder
I'm having a hard time telling from the documentation exactly how to use this module in a scenario where the package is in a local yum repository rather than served by the puppetmaster, but I've tried to work it out from the source code. I'm not clear on why I'm getting this particular error though.
Puppet: 4.10.6
Ruby: 2.0.0p648
Distribution: RedHat
Module version: 7.2.0
splunk_server { 'clustering/pass4SymmKey':
value => "password",
}
The password is set and splunk automatically encrypts it using its splunk.secret
, however splunk and puppet then begin fighting over the password line, with puppet changing it back to the unencrypted password and restarting splunk (and then splunk changing it back to the encrypted password).
The password to be set and not keep changing back and forth, and splunk to not restart every run.
Unfortunately, it seems that splunk attempts to hold their method of encrypting files very close, so implementing the encryption of the file inside of puppet may be difficult. I have had a few thoughts about generating a shasum of the password and using that to verify the password hasn't changed since last run, but even that is difficult since technically the server.secret can change and the password would then not be updated accordingly (unless puppet is also managing the server.secret).
2016.4.0
ruby 2.1.9p490 (2016-03-30 revision 54437) [x86_64-linux]
ce5b15ad5993cae6b72739d9f05f656ebfc5347c
There are defined resource types for all of the other splunkforwarder_*
configurations. However, we need one that handles limits and is placed in /opt/splunkforwarder/etc/system/local/limits.conf
. It appears this one has just been missed.
It should behave exactly the same as all of the other splunkforwarder_*
defined resource types.
The other defined resource types work great.
Currently, the splunk::password needs to also accept an $sslconfig parameter which needs to also be distributed to the forwarders /opt/splunkforwarder/etc/system/local/server.conf. Currently this is not being done and essentially when you use this class to distribute your splunk admin credentials the splunk forwarder fails to run afterwards because of ssl failing in regards to decryption/encryption
class { '::splunk::params':
version => '6.6.1',
build => 'aeae3fe0c5af',
server => 'splunk.acme.local',
}
include ::splunk
Puppet run throws the following error:
Failed to apply catalog: Parameter path failed on File[/opt/splunk/etc/system/local/authentication.conf]: File paths must be fully qualified, not '/opt/splunk/etc/system/local/authentication.conf' at /etc/puppetlabs/code/environments/production/modules/splunk/manifests/init.pp:221
Looks like the module maybe isn't intended to install Splunk (versus Forwarder) in Windows?
Puppet code:
include '::splunk'
include '::splunk::forwarder'
Hiera value:
splunk::params::version: '6.5.0'
splunk::params::build: '59c8927def0f'
splunk::params::src_root: 's3://rg-infrastructure/splunk_install'
splunk::params::server: 'siem'
Error message:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Splunk::Platform::Posix] is already declared; cannot redeclare at /var/lib/rg_data/puppet/environments/production/modules/splunk/manifests/forwarder.pp:133 on node siem
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
No error message
The splunk::addon defined type uses the the concat type from the puppetlabs/concat puppet module. This should be added as a dependency in the metadata.json file
class { '::splunk::forwarder': }
@splunkforwarder_input { 'source_type_example':
section => "monitor://$logs_path/*.log",
setting => 'sourcetype',
value => $name,
tag => "splunk_forwarder"
}
When applying a change to an instance that has already had puppet applied to it (eg. splunkforwarder already installed, then editing 'source_type_example' and running again), splunkforwarder is not restarted.
When making a chance to the resource type for splunk, splunk should be restarted
Notice: /Stage[main]/Splunk::Forwarder/File[/opt/splunkforwarder/etc/system/local/inputs.conf]/ensure: created
Notice: /Stage[main]/Splunk::Forwarder/Splunkforwarder_input[default_host]/ensure: created
No mention of a restart in the output, when running splunk list monitors
the new input doesn't exist. Once I manually restart splunk splunk restart
, the monitor then shows.
@@splunk_server { 'clustering/master_uri':
value => "https://${::fqdn}:8089",
}
The resource is automatically realized on the node it is declared on.
According to the documentation on exported resources it should not be realized until it is collected, however even with no collectors declared in my code it is automatically being collected.
Declaring an exported resource causes that resource to be added to the catalog and marked with an “exported” flag, which prevents Puppet agent from managing the resource (unless it was collected).
Notice: /Stage[main]/Profile::Splunk::Cluster_master/Splunk_server[clustering/master_uri]/ensure: created
This is an enhancement request.
For those using Satellite or a package manager with the splunkforwarder in their repos it would be great to be able to install the splunkforwarder with the package resource instead of bundling it with the module/in Puppet repo.
Any reason this is not supported?
Try to add a monitor resource without any option in the inputs.conf (or with an empty value)
The corresponding line is not added
The corresponding line should be added
Not working:
splunk::forwarder::forwarder_input :
'default_foo':
section : 'monitor://foo/bar.log'
tag : 'splunk_forwarder'
Not working:
splunk::forwarder::forwarder_input :
'default_foo':
section : 'monitor://foo/bar.log'
setting : 'foo'
value :
tag : 'splunk_forwarder'
Working (but I don't want to declare any variable for this monitor entry) :
splunk::forwarder::forwarder_input :
'default_foo':
section : 'monitor://foo/bar.log'
setting : 'foo'
value : ''
tag : 'splunk_forwarder'
Working (but I don't want to declare any variable for this monitor entry):
splunk::forwarder::forwarder_input :
'default_foo':
section : 'monitor://foo/bar.log'
setting : 'foo'
value : 'bar'
tag : 'splunk_forwarder'
Include splunk or splunk::forwarder class on a system
[root@splunk ~]# puppet agent --test --noop
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Redefining rhsm_register in Puppet::Type
Info: Redefining augeasprovider in Puppet::Type
Info: Redefining shellvar in Puppet::Type
Info: Redefining datacat_fragment in Puppet::Type
Info: Redefining ini_setting in Puppet::Type
Info: Redefining firewall in Puppet::Type
Info: Redefining firewallchain in Puppet::Type
Info: Redefining foreman_resource in Puppet::Type
Info: Redefining idmapd_config in Puppet::Type
Info: Redefining gpg_key in Puppet::Type
Info: Redefining rhsm_repo in Puppet::Type
Warning: The dns_rr resource type is deprecated. Use resource_record instead
Info: Redefining file_line in Puppet::Type
Warning: The a2mod provider is deprecated, please use apache::mod instead
Info: Redefining anchor in Puppet::Type
Info: Redefining datacat_collector in Puppet::Type
Info: Caching catalog for splunk.infra.osc.edu
Info: Applying configuration version '1486415397'
Output produced by commenting out profile that includes splunk class and resources.
[root@splunk ~]# puppet agent --test --noop
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for splunk.infra.osc.edu
Info: Applying configuration version '1486415694'
Define a splunkforwarder_limits resource anywhere.
Error: /Stage[main]/Profile_universalforwarder::Config/Splunkforwarder_limits[inputproc max_fd]: Could not evaluate: file_path must be set with splunk_config type before provider can be used
No error
Error: /Stage[main]/Profile_universalforwarder::Config/Splunkforwarder_limits[inputproc max_fd]: Could not evaluate: file_path must be set with splunk_config type before provider can be used
Any other splunkforwarder_* resource is fine.
splunkforwarder_limits seems to be missing from the set_provider_paths function in type/splunk_config.rb around line 115.
class { 'splunk::params':
server => 'splunk.foo.net',
}
class { 'splunk::forwarder':
package_source => '\\foo.net\dfs\splunk_forwarder\splunkforwarder-6.3.3-f44afce176d0-x64-release.msi'
}
PS C:\Users\natemccurdy> puppet agent -t --environment splunk_agent_install --noop
Info: Using configured environment 'plunk_agent_install'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Redefining firewalld_direct_rule in Puppet::Type
Warning: C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet/type/network_config.rb:6: ipaddress gem was not found
Error: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper
Error: Could not autoload puppet/type/splunk_config: Could not autoload puppet/provider/network_config/interfaces: canno
t load such file -- puppetx/filemapper
Error: Could not retrieve catalog from remote server: Could not intern from text/pson: Could not autoload puppet/type/sp
lunk_config: Could not autoload puppet/provider/network_config/interfaces: cannot load such file -- puppetx/filemapper
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Splunk Forwarder MSI installed and configured with no error messages.
This is a Ruby load issue that looks awfully familiar to #86
Actually, looking at it now, it seems that there just needs to be a new module version cut that includes #95 in it.
The splunkd_port needs to not be used for a splunk forwarder.
Hi,
Im unable to figure out how to set the indexer that this puppet-splunk module will send data to.... I've got a splunk server with a "LINUX" index - how can I configure the forwarder to send to that index rather than default? Im using the universal fwder 7.2.0
And while Im at it, thank you for a great module - this has saved me so much time!
-Matt
Error: Could not autoload puppet/type/splunk_config: /var/lib/puppet/lib/puppet/type/splunk_config.rb:7: syntax error, unexpected ':', expecting ')'
newparam(:name, namevar: true) do
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:128: syntax error, unexpected ':', expecting ')'
name: instance.name,
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:129: syntax error, unexpected ':', expecting '='
section: instance[:section],
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:130: syntax error, unexpected ':', expecting '='
setting: instance[:setting],
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:131: syntax error, unexpected kENSURE, expecting '='
ensure: :absent
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:132: syntax error, unexpected ')', expecting kEND
Error: Could not retrieve catalog from remote server: Could not intern from text/pson: Could not autoload puppet/type/splunk_config: /var/lib/puppet/lib/puppet/type/splunk_config.rb:7: syntax error, unexpected ':', expecting ')'
newparam(:name, namevar: true) do
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:128: syntax error, unexpected ':', expecting ')'
name: instance.name,
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:129: syntax error, unexpected ':', expecting '='
section: instance[:section],
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:130: syntax error, unexpected ':', expecting '='
setting: instance[:setting],
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:131: syntax error, unexpected kENSURE, expecting '='
ensure: :absent
^
/var/lib/puppet/lib/puppet/type/splunk_config.rb:132: syntax error, unexpected ')', expecting kEND
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
To be installed smoothly
I have installed the same module on both centos 6 and 7 but with different ruby version (2.1) and it worked fine.
Hello,
I'm attempting to use your module to deploy the Splunk Forwarder only and Puppet throws a dependancy cycle error. I'm testing using Pupply apply if that makes a differenance.
Notice: Compiled catalog for <hostname> in environment production in 0.65 seconds
Error: Failed to apply catalog: Found 1 dependency cycle:
(File[/opt/staging/splunk/splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm] => Staging::File[splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm] => Package[splunkforwarder] => File[/opt/staging/splunk/splunkforwarder-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm])
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
Here is how I call the module:
class profiles::splunk_forwarder (
$server = 'splunk.domain',
$version = '6.5.0',
$build = '59c8927def0f',
) {
class { ::splunk::params:
server => "$server",
version => "$version",
build => "$build"
}
include ::splunk::forwarder
}
Deploy splunk::forwarder to windows with default options
on lines in params
These default paths will NEVER let the MSI actually install using the package manager.
as these are directly included into install arguments. please see this note on puppet docs on Package resource:
With Windows packages, note that file paths in an install option must use backslashes. (Since install options are passed directly to the installation command, forward slashes won’t be automatically converted like they are in file resources.) Note also that backslashes in double-quoted strings must be escaped and backslashes in single-quoted strings can be escaped.
if $::osfamily == 'Windows' {
$forwarder_dir = pick($forwarder_installdir, 'C:/Program Files/SplunkUniversalForwarder')
$server_dir = pick($server_installdir, 'C:/Program Files/Splunk')
} else {
$forwarder_dir = pick($forwarder_installdir, '/opt/splunkforwarder')
$server_dir = pick($server_installdir, '/opt/splunk')
}
Heads up... I've not yet tracked down all of these issues here but if you classify as such:
class { 'splunk::linux_forwarder':
splunk_ver = <some non-default value>,
}
you won't get the version specified due to the way the default version is specified in params and set in the implementation class via inheritance. I believe that the use of way inheritance is used in this module may mean there are other similar issues.
Install latest version of puppetlabs/inifile module
> puppet module install puppetlabs-inifile --version 2.2.0
Install latest version of puppet/splunk module
> puppet module install puppet-splunk --version 7.1.0
Install
I see the following in my puppet server's logs
2018-03-08 05:19:15,307 WARN [qtp1770542989-657360] [puppetserver] Puppet ModuleLoader: module 'splunk' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
(file & line not available)
Error: Could not install module 'puppet-splunk' (???)
No version of 'puppet-splunk' can satisfy all dependencies
Use `puppet module install --ignore-dependencies` to install only this module
Failed to install: the installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid windows installers package.
the installer wasn't able to be download from the puppet master.
When trying to use splunkforwarder_server type i get the following error on both ubuntu 16.04 and windows 2016
Error: /Stage[main]/Profile::Linux::Splunk::Universalforwarder/Splunkforwarder_server[sslConfig/sslPassword]: Could not evaluate: file_path must be set with splunk_config type before provider can be used
Notice: /Stage[main]/Splunk::Virtual/Service[splunk]: Dependency Splunkforwarder_server[sslConfig/sslPassword] has failures: true
Warning: /Stage[main]/Splunk::Virtual/Service[splunk]: Skipping because of failed dependencies
Here is the code i'm using
splunkforwarder_server { 'sslConfig/sslPassword':
value => '$1$Bj3FSJRCLtSH',
tag => 'splunk_forwarder'
}
It would be nice to have the ability to specify a ui-prefs.conf in order to change the default time range of every user's search to prevent causing inefficient queries on accident and remembering to change the time range for every new search
class mailonline::splunk2 {
class { '::splunk::params':
version => '6.6.3',
src_root => 'http://somebucket.s3-website-eu-west-1.amazonaws.com',
splunkd_port => '8089',
}
include ::splunk
}
Error: Failed to apply catalog: no parameter named 'section'
Notice: Applied catalog in n.nn seconds
Info: Using configured environment 'env'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: Creating splunk_index via Puppet.newtype is deprecated and will be removed in a future release. Use Puppet::Type.newtype instead.
(at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:162:in `newtype')
Info: Caching catalog for
Error: Failed to apply catalog: no parameter named 'section'
Tried it on v6.3.1 as well same issue
class profiles::splunkforwarder
{
$version = hiera('splunk::version', '6.5.0')
$build = hiera('splunk::build', '59c8927def0f')
$src_root = hiera('splunk::src_root')
$my_splunk_server = hiera('splunk::deploymentserver')
class { '::splunk::params':
server => $my_splunk_server,
version => $version,
build => $build,
src_root => $src_root,
}
include ::splunk::forwarder
When trying to install just the forwarder on a host it errors out on the purge_* parameters saying they don't exist, even though they are defined in forwarder.pp
Successful puppet run with the splunk forwarder installed, configured and running.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'purge_forwarder_deploymentclient' at /etc/puppetlabs/code/environments/karen/modules/splunk/manifests/params.pp:247 on Splunk_config[splunk] at /etc/puppetlabs/code/environments/karen/modules/splunk/manifests/params.pp:247 on node pup-karen.corp.cira.ca
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
When I comment out the following section from the forwarder.pp file, it functions as expected:
Splunk_config['splunk'] {
purge_forwarder_deploymentclient => $purge_deploymentclient,
purge_forwarder_outputs => $purge_outputs,
purge_forwarder_inputs => $purge_inputs,
purge_forwarder_props => $purge_props,
purge_forwarder_transforms => $purge_transforms,
purge_forwarder_web => $purge_web,
}
class { '::splunk::params':
server => $my_splunk_server,
}
include ::splunk::forwarder
A normal upgrade of the forwarder will not trigger the license_splunkforwarder exec resource.
The license_splunkforwarder exec resource should be triggered any time the forwarder package is installed. That should include initial installs, as well as any upgrades.
Every time the splunk forwarder package is installed, it causes the service to return to a "first time run" state. This means that the license agreement has to be agreed to before the service will start successfully.
Triggering the license_splunkforwarder exec resource off the existence of the server.pem file exists prevents the exec resource from running again, unless someone decides they want to delete the server.pem file.
Installing splunk or the forwarder always creates the ftr
file. Once the license agreement has been accepted the ftr
file is removed. This should make for a better trigger for the license_splunkforwarder exec resource. This way, the license gets accepted any time the package is installed.
Currently this module does not work when installing splunk server on windows O/S. Lets get this feature working!
Here is a bug that was reported because of this feature missing. #115
class profile::splunk_forwarder {
include ::splunk::forwarder
}
splunk::params::src_root: 'puppet:///software'
splunk::params::version: '6.5.1'
splunk::params::build: 'f74036626f0c'
splunk::forwarder::forwarder_output:
'tcpout_defaultgroup':
section: 'default'
setting: 'defaultGroup'
value: 'default-autolb-group'
tag: 'splunk_forwarder'
'tcpout_default-autolb-group':
section: 'tcpout:default-autolb-group'
setting: 'server'
value: ''
tag: 'splunk_forwarder'
Error: Failed to apply catalog: Found 1 dependency cycle:
(File[/opt/staging/splunk/splunkforwarder-6.5.1-f74036626f0c-linux-2.6-x86_64.rpm] => Staging::File[splunkforwarder-6.5.1-f74036626f0c-linux-2.6-x86_64.rpm] => Package[splunkforwarder] => File[/opt/staging/splunk/splunkforwarder-6.5.1-f74036626f0c-linux-2.6-x86_64.rpm])
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
Splunk Forwarder should install from the software location served from my puppet master and then outputs should be configured.
class { '::splunk::params':
version => '=7.1.1',
build => '8f0ead9ec3db',
}
class { '::splunk::forwarder':
pkg_provider => 'yum',
splunk_user => 'splunk',
}
Notice: /Stage[main]/Splunk::Forwarder/Package[splunkforwarder]/ensure: created
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: Password must contain at least:
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: * 8 total printable ASCII character(s).
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns:
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: This appears to be your first time running this version of Splunk.
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns:
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: An Admin password must be set before installation proceeds.
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: tcgetattr: Inappropriate ioctl for device
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: WARNING: error changing terminal modes - password will echo!
Notice: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: Please enter a new password:
Error: 'splunk start --accept-license --answer-yes' returned 1 instead of one of [0]
Error: /Stage[main]/Splunk::Platform::Posix/Exec[license_splunkforwarder]/returns: change from notrun to 0 failed: 'splunk start --accept-license --answer-yes' returned 1 instead of one of [0]
A successful install and configuration of the splunkforwarder.
Splunk Enterprise 7.1 introduces a new password scheme for Splunk software users. This scheme includes additional settings and configuration options, which can affect how you upgrade if you use scripts to automate the upgrade process. You might need to change your upgrade scripts before performing scripted upgrades. Specifically, confirm that you do not pass any illegal arguments to the Splunk CLI for starting or restarting Splunk Enterprise during the upgrade, as this could result in a situation where Splunk Enterprise does not start after the upgrade has completed.
From http://docs.splunk.com/Documentation/Splunk/7.1.1/Installation/AboutupgradingREADTHISFIRST
I set the user under class ::splunk::forwarder to splunk but still the spunkd.pid runs as root, the customer wants this running as splunk, I made a file resource that recurse /opt/splunkforwarder to run as group and owner root, but the change happens every run with a bunch of others /opt/splunkforwarder/var/libsplunk and /opt/splunkforwarder/var/run/*
Manually the customer set chown /opt/forwarder/* to run as splunk then restarted the service and this worked
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/apps/learned/local/props.conf]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/apps/learned/local/props.conf]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/apps/learned/local/sourcetypes.conf]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/apps/learned/local/sourcetypes.conf]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/apps/learned/metadata/local.meta]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/apps/learned/metadata/local.meta]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/passwd]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/etc/passwd]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/4069420869]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/4069420869]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/1322324208]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/1322324208]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/1322324208.old]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/1322324208.old]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/2342085248]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/2342085248]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/2342085248.old]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/lib/splunk/fishbucket/rawdata/2342085248.old]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/run/splunk/conf-mutator.pid]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/run/splunk/conf-mutator.pid]/group: group changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/run/splunk/splunkd.pid]/owner: owner changed 'root' to 'splunk'
Notice: /Stage[main]/Profile::Linux_splunk/File[/opt/splunkforwarder/var/run/splunk/splunkd.pid]/group: group changed 'root' to 'splunk'
Notice: Applied catalog in 2.94 seconds
On every run
My profile
#Linux profile to install Linux Splunk
class profile::linux_splunk {
$artifactory_host = hiera('artifactory_host')
if $facts['apache_version'] {
$client_name = 'fspptuxapch'
acl { '/app/httpd/log/':
action => set,
permission => [
'group:splunk:r-x',
'default:group:splunk:r-x',
],
provider => posixacl,
recursive => true,
require => Package['splunkforwarder'],
}
}
else { $client_name = 'fspptux'
}
splunkforwarder_deploymentclient { 'deployment-client-disabled':
section => 'deployment-client',
setting => 'disabled',
value => '0',
}
splunkforwarder_deploymentclient { 'deployment-client-client-name':
section => 'deployment-client',
setting => 'clientName',
value => $client_name,
}
splunkforwarder_deploymentclient { 'deployment-server':
section => 'target-broker:deploymentServer',
setting => 'targetUri',
value => Blank:8089',
}
class { '::splunk::params':
version => '6.5.2',
build => '67571ef4b87d',
src_root => "http://${artifactory_host}/artifactory/application-release-local/gov/usda/fs/busops/cio/FS_Splunk",
server => 'blank..com',
splunkd_port => '8089',
}
class { '::splunk::forwarder':
splunk_user => 'splunk',
}
file {'/opt/splunkforwarder/etc/splunk-launch.conf':
ensure => present,
owner => 'root',
group => 'splunk',
recurse => false,
require => Package['splunkforwarder'],
}
file {'/opt/splunkforwarder/':
ensure => present,
owner => 'splunk',
group => 'splunk',
recurse => true,
ignore => '/opt/splunkforwarder/etc/splunk-launch.conf',
before => File['/opt/splunkforwarder/etc/splunk-launch.conf'],
}
file {'/var/log':
ensure => present,
owner => 'root',
group => 'root',
mode => "2755",
}
acl { '/var/log':
action => set,
permission => [
'group:splunk:r-x',
'default:group:splunk:r-x',
],
provider => posixacl,
recursive => true,
require => Package['splunkforwarder'],
}
include ::splunk::forwarder
}
This works on windows server 2016 along with ubuntu 16.04
Splunk allows downloading of the splunkserver and forwarder via their https://download.splunk.com url. This module does already allow downloading over https but since splunk change the http path structure convention for downloading their installers this module never got updated.
This would also allow for running automated acceptance tests on travis.
class { '::splunk::params':
server => hiera(my_splunk_server),
version => hiera(my_splunk_version),
build => hiera(my_splunk_build),
src_root => hiera(my_splunk_src_root),
}
class { '::splunk::forwarder':
}
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Unknown variable: 'splunk::params::forwarder_install_options'.
Compile without error
Need to put forwarder_install_options
as class parameter for params
and assign undef
as default value or assign undef
for different kernels other than Windows to avoid this compile error.
It would be awesome to be able to manage a deployment servers serverclass.conf in order to allow puppet to create server classes in an automated fashion.
Try to enable splunk::forwarder::purge_inputs
splunk::forwarder::purge_inputs : true
An error message is generated during the provision:
Notice: Compiled catalog for puppet in environment production in 4.92 seconds
Error: /Stage[main]/Splunk::Params/Splunk_config[splunk]: Failed to generate additional resources using 'generate': can't convert nil into String
Notice: Finished catalog run in 15.65 seconds
No error message
class { '::splunk::params':
version => $version,
build => $build,
src_root => $src_root,
}
include ::splunk::forwarder
puppet agent -t
Error 500 on agent
Forwarder installed.
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'purge_forwarder_deploymentclient' at /etc/puppetlabs/code/environments/master/modules/splunk/manifests/params.pp:258 on Splunk_config[splunk] at /etc/puppetlabs/code/environments/master/modules/splunk/manifests/params.pp:258 on node XXX
I have been using puppet-splunk version 5.1.0 so far.
After upgrading to version 7.0.0 I get following error and can't install forwarder any more.
Hello SE Team,
I'm using this new version of your module and getting the following very unhelpful, unspecific errors.
Error: /Stage[main]/Splunk::Forwarder/Splunkforwarder_output[tcpout_defaultgroup]: Could not evaluate: Invalid parameter key_val_separator(:key_val_separator)
Error: /Stage[main]/Splunk::Forwarder/Splunkforwarder_input[default_host]: Could not evaluate: Invalid parameter key_val_separator(:key_val_separator)
Error: /Stage[main]/Splunk::Forwarder/Splunkforwarder_output[defaultgroup_server]: Could not evaluate: Invalid parameter key_val_separator(:key_val_separator)
Error: /Stage[main]/Splunk::Platform::Posix/Splunkforwarder_input[monitor_varlog]: Could not evaluate: Invalid parameter key_val_separator(:key_val_separator)
I've checked my parameters and looked over your module and can't find anything glaringly obvious that would be causing this.
A Google search doesn't really show much. Hope you can help.
Thanks,
Alex
Install both splunk forwarder and splunk server
The install is ok but the corresponding services are not created properly
There is only one service in /etc/init.d with the name splunk and it corresponds to the first service that has been installed during the installation (so it might be splunk-forwarder or splunk-server depending on the order you call the 2 classes)
In my case every times I changed configuration files related to splunk server, this was the splunk forwarder which was restarted (takes me ages to understand why my new settings werent taken into account)
Puppet should manage the 2 services independently : one to restart splunk-forwarder and one to restart splunk-server
The corresponding resources declare the same file so that couldn't work
@exec { 'enable_splunkforwarder':
# The path parameter can't be set because the boot-start silently fails on systemd service providers
command => "${splunk::params::forwarder_dir}/bin/splunk enable boot-start -user ${splunk_user}",
creates => '/etc/init.d/splunk',
require => Exec['license_splunkforwarder'],
tag => 'splunk_forwarder',
notify => Service['splunk'],
}
When the first exec has run, the second exec below is never executed as the '/etc/init.d/splunk' file already exists.
@exec { 'enable_splunk':
# The path parameter can't be set because the boot-start silently fails on systemd service providers
command => "${splunk::params::server_dir}/bin/splunk enable boot-start -user ${splunk_user}",
creates => '/etc/init.d/splunk',
require => Exec['license_splunk'],
tag => 'splunk_server',
}
Setting purge_inputs and purge_outputs to true is not actually purging unmanaged settings in those files. I have reproduced this on our forwarder on windows. We still need to investigate if this is O/S specific and also what all purging is actually broken and fix it.
This is an enhancement request.
As part of Splunk Enterprise best practices Splunk should run as 'non-root user'. See official splunk documentation http://docs.splunk.com/Documentation/Splunk/7.0.0/Installation/RunSplunkasadifferentornon-rootuser
} else {
$forwarder_dir = pick($forwarder_installdir, '/opt/splunkforwarder')
$server_dir = pick($server_installdir, '/opt/splunk')
$splunk_user = 'root'
}
splunk runs as "root" user by default. Which is against splunk best practices.
The module should create a non-root user (i.e "splunk" user) and run splunk as it.
Run unit tests on role class that pulls in custom types from the splunk module.
This error:
Error while evaluating a Resource Statement, Could not autoload puppet/type/splunk_config: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/splunk_authentication'.
Full output:
error during compilation: Evaluation Error: Error while evaluating a Resource Statement, Could not autoload puppet/type/splunk_config: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/splunk_authentication'. Originally set at file:/Users/tdockendorf/puppet/osc-puppetmaster-conf/modules/splunk/lib/puppet/type/splunk_authentication.rb?line=3. at /Users/tdockendorf/puppet/osc-puppetmaster-conf/modules/splunk/manifests/params.pp:247:3 on node osc-nc167012.local
Unit tests to pass
The unit tests work when I use ruby-2.0.0 and Puppet 3.8.6 with future parser. It's not until I switch to ruby-2.4.0 and Puppet 5.3.2 that things begin to fail.
I have two roles, one for search servers (web + search node) and one for indexers. The unit tests fail for both with same error. Unit tests for my roles that use the splunk forwarder types also fail in same way.
I have not tried the failing combination on real systems yet.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.