Giter VIP home page Giter VIP logo

puppet-filebeat's Introduction

puppet-filebeat

Build Status

Table of Contents

Description

The filebeat module installs and configures the filebeat log shipper maintained by elastic.

Setup

What filebeat affects

By default filebeat adds a software repository to your system, and installs filebeat along with required configurations.

Upgrading to Filebeat 7.x

To upgrade to Filebeat 7.x, simply set $filebeat::major_version to 7 and $filebeat::package_ensure to latest (or whichever version of 7.x you want, just not present).

You'll also need to change instances of filebeat::prospector to filebeat::input when upgrading to version 4.x of this module.

Setup Requirements

The filebeat module depends on puppetlabs/stdlib, and on puppetlabs/apt on Debian based systems.

Beginning with filebeat

filebeat can be installed with puppet module install pcfens-filebeat (or with r10k, librarian-puppet, etc.)

The only required parameter, other than which files to ship, is the outputs parameter.

Usage

All of the default values in filebeat follow the upstream defaults (at the time of writing).

To ship files to elasticsearch:

class { 'filebeat':
  outputs => {
    'elasticsearch' => {
     'hosts' => [
       'http://localhost:9200',
       'http://anotherserver:9200'
     ],
     'loadbalance' => true,
     'cas'         => [
        '/etc/pki/root/ca.pem',
     ],
    },
  },
}

To ship log files through logstash:

class { 'filebeat':
  outputs => {
    'logstash'     => {
     'hosts' => [
       'localhost:5044',
       'anotherserver:5044'
     ],
     'loadbalance' => true,
    },
  },
}

Shipper and logging options can be configured the same way, and are documented on the elastic website.

Adding an Input

Inputs are processes that ship log files to elasticsearch or logstash. They can be defined as a hash added to the class declaration (also used for automatically creating input using hiera), or as their own defined resources.

At a minimum, the paths parameter must be set to an array of files or blobs that should be shipped. doc_type is what logstash views as the type parameter if you'd like to apply conditional filters.

filebeat::input { 'syslogs':
  paths    => [
    '/var/log/auth.log',
    '/var/log/syslog',
  ],
  doc_type => 'syslog-beat',
}

Multiline Logs

Filebeat inputs can handle multiline log entries. The multiline parameter accepts a hash containing pattern, negate, match, max_lines, and timeout as documented in the filebeat configuration documentation.

JSON Logs

Filebeat inputs (versions >= 5.0) can natively decode JSON objects if they are stored one per line. The json parameter accepts a hash containing message_key, keys_under_root, overwrite_keys, and add_error_key.

Depending on the version, expand_keys, document_id and ignore_decoding_error may be supported as well.

See the filebeat configuration documentation for details.

Inputs in Hiera

Inputs can be defined in hiera using the inputs parameter. By default, hiera will not merge input declarations down the hiera hierarchy. That behavior can be changed by configuring the lookup_options flag.

inputs can be a Hash that will follow all the parameters listed on this documentation or an Array that will output as is to the input config file.

Usage of filebeat modules

Filebeat ships with modules which contain pipelines and dashboards for common software. Filebeat needs to be setup to ship directly into elasticsearch that it's possible that filebeat will setup pipelines and dashboards automatically.

If your setup includes logstash or some other service between filebeat and elasticsearch the following settings might not work as expected.

The following should be a minimal example to get filebeat::module::* to create the required config and push pipeline and dashboards into your elasticsearch & kibana.

class { 'filebeat::module::system':
  syslog_enabled => true,
  auth_enabled => true,
}

class { 'filebeat':
  enable_conf_modules => true,
  overwrite_pipelines => true,
  setup => {
    dashboards => {
      enabled => true
    },
    kibana => {
      host => 'http://kibana.example.com:5601',
    }
  }
}

Usage on Windows

When installing on Windows, this module will download the windows version of Filebeat from elastic to C:\Temp by default. The directory can be overridden using the tmp_dir parameter. tmp_dir is not managed by this module, but is expected to exist as a directory that puppet can write to.

Processors

Filebeat 5.0 and greater includes a new libbeat feature for filtering and/or enhancing all exported data through processors before being sent to the configured output(s). They can be defined as a hash added to the class declaration (also used for automatically creating processors using hiera), or as their own defined resources.

To drop the offset and input_type fields from all events:

class {'filebeat':
  processors => [
    {
      'drop_fields' => {
        'fields' => ['input_type', 'offset'],
      }
    }
  ],
}

To drop all events that have the http response code equal to 200: input

class {'filebeat':
  processors => [
    {
      'drop_event' => {
        'when' => {'equals' => {'http.code' => 200}}
      }
    }
  ],
}

Now to combine these examples into a single definition:

class {'filebeat':
  processors => [
    {
      'drop_fields' => {
        'params'   => {'fields' => ['input_type', 'offset']},
        'priority' => 1,
      }
    },
    {
      'drop_event' => {
        'when'     => {'equals' => {'http.code' => 200}},
        'priority' => 2,
      }
    }
  ],
}

For more information please review the documentation here.

Processors in Hiera

Processors can be declared in hiera using the processors parameter. By default, hiera will not merge processor declarations down the hiera hierarchy. That behavior can be changed by configuring the lookup_options flag.

Index Lifecycle Management

You can override the default filebeat ILM policy by specifying ilm.policy hash in filebeat::setup parameter:

filebeat::setup:
  ilm.policy:
    phases:
      hot:
        min_age: "0ms"
        actions:
          rollover:
            max_size: "10gb"
            max_age: "1d"

Reference

Public Classes

Class: filebeat

Installs and configures filebeat.

Parameters within filebeat

  • package_ensure: [String] The ensure parameter for the filebeat package If set to absent, inputs and processors passed as parameters are ignored and everything managed by puppet will be removed. (default: present)
  • manage_package: [Boolean] Whether ot not to manage the installation of the package (default: true)
  • manage_repo: [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
  • major_version: [Enum] The major version of Filebeat to install. Should be either '5' or '6'. The default value is '6', except for OpenBSD 6.3 and earlier, which has a default value of '5'.
  • service_ensure: [String] The ensure parameter on the filebeat service (default: running)
  • service_enable: [String] The enable parameter on the filebeat service (default: true)
  • param repo_priority: [Integer] Repository priority. yum and apt supported (default: undef)
  • service_provider: [String] The provider parameter on the filebeat service (default: on RedHat based systems use redhat, otherwise undefined)
  • spool_size: [Integer] How large the spool should grow before being flushed to the network (default: 2048)
  • idle_timeout: [String] How often the spooler should be flushed even if spool size isn't reached (default: 5s)
  • publish_async: [Boolean] If set to true filebeat will publish while preparing the next batch of lines to transmit (default: false)
  • config_file: [String] Where the configuration file managed by this module should be placed. If you think you might want to use this, read the limitations first. Defaults to the location that filebeat expects for your operating system.
  • config_dir: [String] The directory where inputs should be defined (default: /etc/filebeat/conf.d)
  • config_dir_mode: [String] The permissions mode set on the configuration directory (default: 0755)
  • config_dir_owner: [String] The owner of the configuration directory (default: root). Linux only.
  • config_dir_group: [String] The group of the configuration directory (default: root). Linux only.
  • config_file_mode: [String] The permissions mode set on configuration files (default: 0644)
  • config_file_owner: [String] The owner of the configuration files, including inputs (default: root). Linux only.
  • config_file_group: [String] The group of the configuration files, including inputs (default: root). Linux only.
  • purge_conf_dir: [Boolean] Should files in the input configuration directory not managed by puppet be automatically purged
  • enable_conf_modules: [Boolean] Should filebeat.config.modules be enabled
  • modules_dir: [String] The directory where module configurations should be defined (default: /etc/filebeat/modules.d)
  • cloud: [Hash] Will be converted to YAML for the optional cloud.id and cloud.auth of the configuration (see documentation, and above)
  • features: [Hash] Will be converted to YAML for the optional features section of the configuration (see documentation, and above)
  • queue: [Hash] Will be converted to YAML for the optional queue.mem and queue.disk of the configuration (see documentation, and above)
  • outputs: [Hash] Will be converted to YAML for the required outputs section of the configuration (see documentation, and above)
  • shipper: [Hash] Will be converted to YAML to create the optional shipper section of the filebeat config (see documentation)
  • autodiscover: [Hash] Will be converted to YAML for the optional autodiscover section of the configuration (see documentation, and above)`
  • logging: [Hash] Will be converted to YAML to create the optional logging section of the filebeat config (see documentation)
  • systemd_beat_log_opts_override: [String] Will overide the default BEAT_LOG_OPTS=-e. Required if using logging hash on systems running with systemd. required: Puppet 6.1+, Filebeat 7+,
  • modules: [Array] Will be converted to YAML to create the optional modules section of the filebeat config (see documentation)
  • conf_template: [String] The configuration template to use to generate the main filebeat.yml config file.
  • download_url: [String] The URL of the zip file that should be downloaded to install filebeat (windows only)
  • install_dir: [String] Where filebeat should be installed (windows only)
  • tmp_dir: [String] Where filebeat should be temporarily downloaded to so it can be installed (windows only)
  • shutdown_timeout: [String] How long filebeat waits on shutdown for the publisher to finish sending events
  • beat_name: [String] The name of the beat shipper (default: FQDN)
  • tags: [Array] A list of tags that will be included with each published transaction
  • max_procs: [Number] The maximum number of CPUs that can be simultaneously used
  • fields: [Hash] Optional fields that should be added to each event output
  • fields_under_root: [Boolean] If set to true, custom fields are stored in the top level instead of under fields
  • disable_config_test: [Boolean] If set to true, configuration tests won't be run on config files before writing them.
  • processors: [Array] Processors that should be configured.
  • monitoring: [Hash] The monitoring.* components of the filebeat configuration.
  • inputs: [Hash] or [Array] Inputs that will be created. Commonly used to create inputs using hiera
  • setup: [Hash] Setup that will be created. Commonly used to create setup using hiera
  • xpack: [Hash] XPack configuration to pass to filebeat
  • extra_validate_options: [String] Extra command line options to pass to the configuration validation command.
  • overwrite_pipelines: [Boolean] If set to true, filebeat will overwrite existing pipelines.

Private Classes

Class: filebeat::config

Creates the configuration files required for filebeat (but not the inputs)

Class: filebeat::install

Calls the correct installer class based on the kernel fact.

Class: filebeat::params

Sets default parameters for filebeat based on the OS and other facts.

Class: filebeat::repo

Installs the yum or apt repository for the system package manager to install filebeat.

Class: filebeat::service

Configures and manages the filebeat service.

Class: filebeat::install::linux

Install the filebeat package on Linux kernels.

Class: filebeat::install::windows

Downloads, extracts, and installs the filebeat zip file in Windows.

Public Defines

Define: filebeat::input

Installs a configuration file for a input.

Be sure to read the filebeat configuration details to fully understand what these parameters do.

Parameters for filebeat::input

  • ensure: The ensure parameter on the input configuration file. (default: present)
  • paths: [Array] The paths, or blobs that should be handled by the input. (required if input_type is log)
  • containers_ids: [Array] If input_type is docker, the list of Docker container ids to read the logs from. (default: '*')
  • containers_path: [String] If input_type is docker, the path from where the logs should be read from. (default: /var/log/docker/containers)
  • containers_stream: [String] If input_type is docker, read from the specified stream only. (default: all)
  • combine_partial: [Boolean] If input_type is docker, enable partial messages joining. (default: false)
  • cri_parse_flags: [Boolean] If input_type is docker, enable CRI flags parsing from the log file. (default: false)
  • syslog_protocol: [Enum tcp,udp] Syslog protocol (default: udp)
  • syslog_host: [String] Host to listen for syslog messages (default: localhost:5140)
  • exclude_files: [Array] Files that match any regex in the list are excluded from filebeat (default: [])
  • encoding: [String] The file encoding. (default: plain)
  • input_type: [String] where filebeat reads the log from (default: filestream)
  • take_over : [Boolean] Optionally enable take_over when switchting from the deprecated input type log to the new input type filestream. This avoids re-ingesting already logfiles Filebeat already read when switching to filestream. This feature requires Filebeat 8.x.
  • fields: [Hash] Optional fields to add information to the output (default: {})
  • fields_under_root: [Boolean] Should the fields parameter fields be stored at the top level of indexed documents.
  • ignore_older: [String] Files older than this field will be ignored by filebeat (default: ignore nothing)
  • close_older: [String] Files that haven't been modified since close_older, they'll be closed. New modifications will be read when files are scanned again according to scan_frequency. (default: 1h)
  • log_type: [String] (Deprecated - use doc_type) The document_type setting (optional - default: log)
  • doc_type: [String] The event type to used for published lines, used as type field in logstash and elasticsearch (optional - default: log)
  • scan_frequency: [String] How often should the input check for new files (default: 10s)
  • harvester_buffer_size: [Integer] The buffer size the harvester uses when fetching the file (default: 16384)
  • tail_files: [Boolean] If true, filebeat starts reading new files at the end instead of the beginning (default: false)
  • backoff: [String] How long filebeat should wait between scanning a file after reaching EOF (default: 1s)
  • max_backoff: [String] The maximum wait time to scan a file for new lines to ship (default: 10s)
  • backoff_factor: [Integer] backoff is multiplied by this parameter until max_backoff is reached to determine the actual backoff (default: 2)
  • force_close_files: [Boolean] Should filebeat forcibly close a file when renamed (default: false)
  • pipeline: [String] Filebeat can be configured for a different ingest pipeline for each input (default: undef)
  • include_lines: [Array] A list of regular expressions to match the lines that you want to include. Ignored if empty (default: [])
  • exclude_lines: [Array] A list of regular expressions to match the files that you want to exclude. Ignored if empty (default: [])
  • max_bytes: [Integer] The maximum number of bytes that a single log message can have (default: 10485760)
  • tags: [Array] A list of tags to send along with the log data.
  • json: [Hash] Options that control how filebeat handles decoding of log messages in JSON format See above. (default: {})
  • multiline: [Hash] Options that control how Filebeat handles log messages that span multiple lines. See above. (default: {})
  • host: [String] Host and port used to read events for TCP or UDP plugin (default: localhost:9000)
  • max_message_size: [String] The maximum size of the message received over TCP or UDP (default: undef)
  • keep_null: [Boolean] If this option is set to true, fields with null values will be published in the output document (default: undef)
  • include_matches: [Array] Journald input only, A collection of filter expressions used to match fields. The format of the expression is field=value (default: [])
  • seek: [Enum] Journald input only, The position to start reading the journal from (default: undef)
  • index: [String] If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs) (default: undef)
  • publisher_pipeline_disable_host: [Boolean] This disables the "host.name" attribute being added to events. See filebeat input configuration reference (default: false)

Define: filebeat::module

Base resource used to implement filebeat module support in this puppet module and can be useful if you have custom filebeat modules.

Parameters for filebeat::module

  • ensure: The ensure parameter on the module configuration file. (default: present)
  • config: [Hash] Full hash representation of the module configuration

Limitations

This module doesn't load the elasticsearch index template into elasticsearch (required when shipping directly to elasticsearch).

When installing on Windows, there's an expectation that C:\Temp already exists, or an alternative location specified in the tmp_dir parameter exists and is writable by puppet. The temp directory is used to store the downloaded installer only.

Generic template

By default, a generic, open ended template is used that simply converts your configuration into a hash that is produced as YAML on the system. To use a template that is more strict, but possibly incomplete, set conf_template to filebeat/filebeat.yml.erb.

Debian Systems

Filebeat 5.x and newer requires apt-transport-https, but this module won't install it for you.

Using config_file

There are a few very specific use cases where you don't want this module to directly manage the filebeat configuration file, but you still want the configuration file on the system at a different location. Setting config_file will write the filebeat configuration file to an alternate location, but it will not update the init script. If you don't also manage the correct file (/etc/filebeat/filebeat.yml on Linux, C:/Program Files/Filebeat/filebeat.yml on Windows) then filebeat won't be able to start.

If you're copying the alternate config file location into the real location you'll need to include some metaparameters like

file { '/etc/filebeat/filebeat.yml':
  ensure  => file,
  source  => 'file:///etc/filebeat/filebeat.special',
  require => File['filebeat.yml'],
  notify  => Service['filebeat'],
}

to ensure that services are managed like you might expect.

Logging on systems with Systemd and with version filebeat 7.0+ installed

With filebeat version 7+ running on systems with systemd, the filebeat systemd service file contains a default that will ignore the logging hash parameter

Environment="BEAT_LOG_OPTS=-e`

to overide this default, you will need to set the systemd_beat_log_opts_override parameter to empty string

example:

class {'filebeat':
  logging => {
    'level'     => 'debug',
    'to_syslog' => false,
    'to_files'  => true,
    'files'     => {
      'path'        => '/var/log/filebeat',
      'name'        => 'filebeat',
      'keepfiles'   => '7',
      'permissions' => '0644'
    },
  systemd_beat_log_opts_override => "",
}

this will only work on systems with puppet version 6.1+. On systems with puppet version < 6.1 you will need to systemctl daemon-reload. This can be achived by using the camptocamp-systemd

include systemd::systemctl::daemon_reload

class {'filebeat':
  logging => {
...
    },
  systemd_beat_log_opts_override => "",
  notify  => Class['systemd::systemctl::daemon_reload'],
}

Development

Pull requests and bug reports are welcome. If you're sending a pull request, please consider writing tests if applicable.

puppet-filebeat's People

Contributors

amitdaniel86-zz avatar axier avatar buzzdeee avatar cdenneen avatar corey-hammerton avatar cristianocasella avatar csoleimani avatar dubuc avatar edestecd avatar ericseiffert avatar farlerac avatar fbarbeira avatar gnobironts avatar jbehrends avatar joshuaspence avatar kpankonen avatar lampapetrol avatar lorenzbausch avatar miranovy avatar mithr4ndir avatar ml14tlc avatar pcfens avatar ralfbosz avatar setaou avatar smokris avatar tmu-sprd avatar trenta avatar trunet avatar tuningyourcode avatar yurymuski avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-filebeat's Issues

Invalid options in output->logstash section of filebeat5.yml.erb

The supported_protocols does not appear to be a valid option in the ssl configuration section. I am not able to find a reference to this in elastic's documentation for logstash. This section of code prevents module from working when ssl parameters are passed:

==> centos-7.2: Error: Evaluation Error: Error while evaluating a Function Call, Failed to parse template filebeat/filebeat5.yml.erb:
==> centos-7.2: Filepath: /tmp/vagrant-puppet/modules-90cc3165c045baafa183f04f99186257/filebeat/templates/filebeat5.yml.erb
==> centos-7.2: Line: 231

Windows failing configtest

I am using this module, version 0.8.6 and deploy FileBeat to a Windows Server 2012 R2 machine. The init.pp file is as follows:
class { 'filebeat':
outputs => {
'logstash' => {
'hosts' => [
'logstash1a:47001'
],
},
},
}

The deployed FileBeat version is 1.3.1 (386) and the filebeat.yml file contains the following output section:
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts:
- logstash1a:47001

When running filebeat.exe -configtest filebeat.yml I get the following result:
Error Initialising publisher: No outputs are defined. Please define one under the output section.

Changing the filebeat.yml file to
#----------------------------- Logstash output --------------------------------
output:
logstash:
hosts:
- logstash1a:47001
makes the configtest pass.

I have puppet agent version 3.8.7 on the Windows Server and puppet master version 3.8.7 on Ubuntu 14.04.

Instantiate Puppet resources (prospectors)

Hello,

I am using the prospectors parameters from the public class to instantiate the filebeat::prospector.
From what ive seen in the init.pp file you already have the 'create_resources' which means all resource properties/params can be passed in using puppet ENC.

if !empty($prospectors_final) { create_resources('filebeat::prospector', $prospectors_final) }

I try to pass the following HASH in the prospector parameter...
paths: ["/var/log/messages"] doc_type: syslog
and I get the following error.

no implicit conversion of Array into Hash at /etc/puppet/environments/production/modules/filebeat/manifests/init.pp:91

From what I understand the only required parameter for filebeat::prospector is paths which is an ARRAY.

Any idea of what is causing this error?

I tested it without the prospector parameters and everything works fine. The filebeat repo is created and filebeat is installed. The custom filebeat.yml is created with the 'ouputs' and 'shippers' parameters that i've defined. When I add the prospector parameters I get the implicit conversion of Array error.

"Could not enable filebeat" error, though working.

Running Debian 16.04 in Azure, using Puppet Agent (and Server) 4.8.1 and latest Filebeat module from the Forge.

Everything works, but each time the agent runs, there's an error:

Debug: Class[Profiles::Filebeat]: The container Stage[main] will propagate my refresh event
Debug: Prefetching apt_key resources for apt_key
Debug: Executing: '/usr/bin/apt-key adv --list-keys --with-colons --fingerprint --fixed-list-mode'
Debug: Executing: '/bin/systemctl is-active filebeat'
Debug: Executing: '/bin/systemctl is-enabled filebeat'
Debug: Executing: '/bin/systemctl unmask filebeat'
Debug: Executing: '/bin/systemctl enable filebeat'
Error: Could not enable filebeat:
Debug: Executing: '/bin/systemctl is-enabled filebeat'
Error: /Stage[main]/Filebeat::Service/Service[filebeat]/enable: change from false to true failed: Could not enable filebeat:
Debug: Class[Filebeat::Service]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Filebeat/Anchor[filebeat::end]: Dependency Service[filebeat] has failures: true
Warning: /Stage[main]/Filebeat/Anchor[filebeat::end]: Skipping because of failed dependencies

I would be grateful for any assistance.

Cheers,

Adding output TLS feature

Hello and thanks for your great module!

What I am missing is the possibility to easily set the TLS ressources.

Example:

outputs => {
    'logstash' => {
        'hosts' => [
            'host:5044',
        ],
        'tls' => {
            'certificate_authorities' => [
                'puppet:///modules/beats/ssl.ca',
                'puppet:///modules/beats/ssl2.ca',
            ],
            'certificate' => 'puppet:///modules/beats/ssl.crt',
            'certificate_key' => 'puppet:///modules/beats/ssl.key',
        },
    },
},

Thanks!

define hosts (filebeat5.yml.erb:)| prospectors error

Error:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Failed to parse template filebeat/filebeat5.yml.erb:
Filepath: /etc/puppetlabs/code/environments/production/modules/filebeat/templates/filebeat5.yml.erb
Line: 197
Detail: undefined method `each' for nil:NilClass
at /etc/puppetlabs/code/environments/production/modules/filebeat/manifests/config.pp:29:20 on node elk-server.example.local

Hiera :

Filebeat config

filebeat::outputs:
'logstash':
host: 'elk-server.example.local'
port: 6379
index: 'packetbeat'
worker: 1

Any suggestions.

error while reading prospector definition from Hiera

Hi,
I have an error while trying to read prospectors declared in Hiera.
Here is my error:

Failure/Error: create_resources('filebeat::prospector', $prospectors_final)

Puppet::PreformattedError:
Evaluation Error: Error while evaluating a Function Call, no implicit conversion of String into Hash at /root/git/my_filebeat/spec/fixtures/modules/filebeat/manifests/init.pp:142:5

My Hiera definition:

filebeat::prospectors:
  ensure: 'present'
  paths:
    - '/tmp/log/production.abc.log'
    - '/tmp/log/production.xyz.log'
  doc_type: 'log4j'
  multiline:
    pattern: '^[.*'
    negate: 'true'
    match: 'after'

My puppet code:

class { 'filebeat':    
      prospectors_merge => true, #reads prospectors from hiera
    }

I guess the error is in my Hiera yml definition.
How to define multiple prospectors?
Thanks.

Need to fix some params as Integer in Redis output

Please add these lines to the template:

<%- if @filebeat_config['output']['redis']['db'] -%>
<%- @filebeat_config['output']['redis']['db'] = Integer(@filebeat_config['output']['redis']['db']) -%>
<%- end -%>
<%- if @filebeat_config['output']['redis']['db_topology'] -%>
<%- @filebeat_config['output']['redis']['db_topology'] = Integer(@filebeat_config['output']['redis']['db_topology']) -%>
<%- end -%>
<%- if @filebeat_config['output']['redis']['timeout'] -%>
<%- @filebeat_config['output']['redis']['timeout'] = Integer(@filebeat_config['output']['redis']['timeout']) -%>
<%- end -%>
<%- if @filebeat_config['output']['redis']['reconnect_interval'] -%>
<%- @filebeat_config['output']['redis']['reconnect_interval'] = Integer(@filebeat_config['output']['redis']['reconnect_interval']) -%>
<%- end -%>

Else filebeat cannot start:

Aug 10 16:52:04 robert2 filebeat[44273]: Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: unmarshal errors:
Aug 10 16:52:04 robert2 filebeat[44273]: line 15: cannot unmarshal !!str 0 into int
Aug 10 16:52:04 robert2 filebeat[44273]: line 16: cannot unmarshal !!str 1 into int
Aug 10 16:52:04 robert2 filebeat[44273]: line 17: cannot unmarshal !!str 5 into int
Aug 10 16:52:04 robert2 filebeat[44273]: line 18: cannot unmarshal !!str 1 into int. Exiting.

Support additional options in prospector

With the newest version of filebeat (v. 1.1) just out, most people are waiting to use the "multiline" which is not included in the prospector template. There are also quite a few other new options which need to be added.

I list of options from https://www.elastic.co/guide/en/beats/filebeat/1.1/filebeat-configuration-details.html that I see need to be added are:

Of these, the most important to everyone is the multiline support.

Apt repository won't install on ubuntu

I am trying to install on Ubuntu 14.04 and get the following error:

Error: validate_re(): input needs to be a String, not a Hash at /etc/puppet/modules/apt/manifests/key.pp:62 on node XXX

I have the following in my puppet class:

class { 'filebeat':
  outputs => {
    'elasticsearch' => {
     'hosts' => $hosts,
    },
    'index'  => 'packetbeat',
    'cas'    => [
       '/etc/pki/root/ca.pem',
    ],
  },
}

I have version 2.3.0 of puppetlabs-apt and 0.8.7 of filebeats

It looks like filebeat::repo is failing to install the beats repo because the key is not valid.
Any ideas?

indentation for processors not working

This feature (supporting processors) does not seem to work.
90e70af

class{"filebeat":
  processors => [
    {
      "name" => "include_fields",
      "params" => {"fields" => ["cpu"]}
    },
    {
      "name" => "drop_fields",
      "params" => {"fields" => ["cpu.user", "cpu.system"]}
    },
    {
      "name" => "drop_event",
      "when" => {"equals" => {"http.code" => 200}}
    },
  ],
}
### puppet agent -t 
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template filebeat/filebeat5.yml.erb:
  Filepath: /etc/puppet/modules/filebeat/templates/filebeat5.yml.erb
  Line: 87
  Detail: undefined method `each' for nil:NilClass
 at /etc/puppet/modules/filebeat/manifests/config.pp:38 on node ubuntu1404-2.vagrant.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

filebeat: 5.1.2
root@master:/etc/puppet/modules/filebeat# git log -1
commit 4cb659d
Author: Phil Fenstermacher [email protected]
Date: Thu Jan 19 12:03:45 2017 -0500

Increment version to v0.10.2

https://github.com/pcfens/puppet-filebeat/pull/69/files#diff-5f87021e7184ecfed031149f8249971dR87
should not be mandatory for drop_event processor i guess.

Tags variable missing in template

Hello there,

There is an option to configure shipper tags in filebeat.yml.ruby18.erb but not in filebeat.yml.erb. Any specific reason why this is not possible?

Kind Regards,

Stefaan

Configuration file ordering with Ruby > 1.8

Hey Y'all,

It seems that the config file generation for the ruby > 1.8 template sometimes results in inconsistent ordering. We're seeing things like this fairly regularly:

+++ /tmp/puppet-file20160630-11451-1fr4lz1      2016-06-30 13:09:32.743719846 -0700
@@ -14,8 +14,8 @@
       - team_one
   output:
     logstash:
-      port: 5044
       loadbalance: true
+      port: 5044

It's not a huge issue since the filebeat process maintains state just fine, but it would be really nice to be able to guarantee puppet exiting cleanly when no changes are actually being made.

Default value for registry is wrong

The default value for $filebeat::registry_file is set to .filebeat. This is inconsistent with the Filebeat documentation which states that the default registry should be /var/lib/filebeat/registry. When I install Filebeat locally with dpkg, the configuration file does indeed point to /var/lib/filebeat/registry:

################### Filebeat Configuration Example #########################

############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log/*/*.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      paths:
        - /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*

      # Configure the file encoding for reading files with international characters
      # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
      # Some sample encodings:
      #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
      #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
      #encoding: plain

      # Type of the files. Based on this the way the file is read is decided.
      # The different types cannot be mixed in one prospector
      #
      # Possible options are:
      # * log: Reads every line of the log file (default)
      # * stdin: Reads the standard in
      input_type: log

      # Exclude lines. A list of regular expressions to match. It drops the lines that are
      # matching any regular expression from the list. The include_lines is called before
      # exclude_lines. By default, no lines are dropped.
      # exclude_lines: ["^DBG"]

      # Include lines. A list of regular expressions to match. It exports the lines that are
      # matching any regular expression from the list. The include_lines is called before
      # exclude_lines. By default, all the lines are exported.
      # include_lines: ["^ERR", "^WARN"]

      # Exclude files. A list of regular expressions to match. Filebeat drops the files that
      # are matching any regular expression from the list. By default, no files are dropped.
      # exclude_files: [".gz$"]

      # Optional additional fields. These field can be freely picked
      # to add additional information to the crawled log files for filtering
      #fields:
      #  level: debug
      #  review: 1

      # Set to true to store the additional fields as top level fields instead
      # of under the "fields" sub-dictionary. In case of name conflicts with the
      # fields added by Filebeat itself, the custom fields overwrite the default
      # fields.
      #fields_under_root: false

      # Ignore files which were modified more then the defined timespan in the past.
      # In case all files on your system must be read you can set this value very large.
      # Time strings like 2h (2 hours), 5m (5 minutes) can be used.
      #ignore_older: 0

      # Close older closes the file handler for which were not modified
      # for longer then close_older
      # Time strings like 2h (2 hours), 5m (5 minutes) can be used.
      #close_older: 1h

      # Type to be published in the 'type' field. For Elasticsearch output,
      # the type defines the document type these entries should be stored
      # in. Default: log
      #document_type: log

      # Scan frequency in seconds.
      # How often these files should be checked for changes. In case it is set
      # to 0s, it is done as often as possible. Default: 10s
      #scan_frequency: 10s

      # Defines the buffer size every harvester uses when fetching the file
      #harvester_buffer_size: 16384

      # Maximum number of bytes a single log event can have
      # All bytes after max_bytes are discarded and not sent. The default is 10MB.
      # This is especially useful for multiline log messages which can get large.
      #max_bytes: 10485760

      # Mutiline can be used for log messages spanning multiple lines. This is common
      # for Java Stack Traces or C-Line Continuation
      #multiline:

        # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
        #pattern: ^\[

        # Defines if the pattern set under pattern should be negated or not. Default is false.
        #negate: false

        # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
        # that was (not) matched before or after or as long as a pattern is not matched based on negate.
        # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
        #match: after

        # The maximum number of lines that are combined to one event.
        # In case there are more the max_lines the additional lines are discarded.
        # Default is 500
        #max_lines: 500

        # After the defined timeout, an multiline event is sent even if no new pattern was found to start a new event
        # Default is 5s.
        #timeout: 5s

      # Setting tail_files to true means filebeat starts readding new files at the end
      # instead of the beginning. If this is used in combination with log rotation
      # this can mean that the first entries of a new file are skipped.
      #tail_files: false

      # Backoff values define how agressively filebeat crawls new files for updates
      # The default values can be used in most cases. Backoff defines how long it is waited
      # to check a file again after EOF is reached. Default is 1s which means the file
      # is checked every second if new lines were added. This leads to a near real time crawling.
      # Every time a new line appears, backoff is reset to the initial value.
      #backoff: 1s

      # Max backoff defines what the maximum backoff time is. After having backed off multiple times
      # from checking the files, the waiting time will never exceed max_backoff idenependent of the
      # backoff factor. Having it set to 10s means in the worst case a new line can be added to a log
      # file after having backed off multiple times, it takes a maximum of 10s to read the new line
      #max_backoff: 10s

      # The backoff factor defines how fast the algorithm backs off. The bigger the backoff factor,
      # the faster the max_backoff value is reached. If this value is set to 1, no backoff will happen.
      # The backoff value will be multiplied each time with the backoff_factor until max_backoff is reached
      #backoff_factor: 2

      # This option closes a file, as soon as the file name changes.
      # This config option is recommended on windows only. Filebeat keeps the files it's reading open. This can cause
      # issues when the file is removed, as the file will not be fully removed until also Filebeat closes
      # the reading. Filebeat closes the file handler after ignore_older. During this time no new file with the
      # same name can be created. Turning this feature on the other hand can lead to loss of data
      # on rotate files. It can happen that after file rotation the beginning of the new
      # file is skipped, as the reading starts at the end. We recommend to leave this option on false
      # but lower the ignore_older value to release files faster.
      #force_close_files: false

    # Additional prospector
    #-
      # Configuration to use stdin input
      #input_type: stdin

  # General filebeat configuration options
  #
  # Event count spool threshold - forces network flush if exceeded
  #spool_size: 2048

  # Enable async publisher pipeline in filebeat (Experimental!)
  #publish_async: false

  # Defines how often the spooler is flushed. After idle_timeout the spooler is
  # Flush even though spool_size is not reached.
  #idle_timeout: 5s

  # Name of the registry file. Per default it is put in the current working
  # directory. In case the working directory is changed after when running
  # filebeat again, indexing starts from the beginning again.
  registry_file: /var/lib/filebeat/registry

  # Full Path to directory with additional prospector configuration files. Each file must end with .yml
  # These config files must have the full filebeat config part inside, but only
  # the prospector part is processed. All global options like spool_size are ignored.
  # The config_dir MUST point to a different directory then where the main filebeat config file is in.
  #config_dir:

###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  ### Elasticsearch as output
  elasticsearch:
    # Array of hosts to connect to.
    # Scheme and port can be left out and will be set to the default (http and 9200)
    # In case you specify and additional path, the scheme is required: http://localhost:9200/path
    # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
    hosts: ["localhost:9200"]

    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "admin"
    #password: "s3cr3t"

    # Number of workers per Elasticsearch host.
    #worker: 1

    # Optional index name. The default is "filebeat" and generates
    # [filebeat-]YYYY.MM.DD keys.
    #index: "filebeat"

    # A template is used to set the mapping in Elasticsearch
    # By default template loading is disabled and no template is loaded.
    # These settings can be adjusted to load your own template or overwrite existing ones
    #template:

      # Template name. By default the template name is filebeat.
      #name: "filebeat"

      # Path to template file
      #path: "filebeat.template.json"

      # Overwrite existing template
      #overwrite: false

    # Optional HTTP Path
    #path: "/elasticsearch"

    # Proxy server url
    #proxy_url: http://proxy:3128

    # The number of times a particular Elasticsearch index operation is attempted. If
    # the indexing operation doesn't succeed after this many retries, the events are
    # dropped. The default is 3.
    #max_retries: 3

    # The maximum number of events to bulk in a single Elasticsearch bulk API index request.
    # The default is 50.
    #bulk_max_size: 50

    # Configure http request timeout before failing an request to Elasticsearch.
    #timeout: 90

    # The number of seconds to wait for new events between two bulk API index requests.
    # If `bulk_max_size` is reached before this interval expires, addition bulk index
    # requests are made.
    #flush_interval: 1

    # Boolean that sets if the topology is kept in Elasticsearch. The default is
    # false. This option makes sense only for Packetbeat.
    #save_topology: false

    # The time to live in seconds for the topology information that is stored in
    # Elasticsearch. The default is 15 seconds.
    #topology_expire: 15

    # tls configuration. By default is off.
    #tls:
      # List of root certificates for HTTPS server verifications
      #certificate_authorities: ["/etc/pki/root/ca.pem"]

      # Certificate for TLS client authentication
      #certificate: "/etc/pki/client/cert.pem"

      # Client Certificate Key
      #certificate_key: "/etc/pki/client/cert.key"

      # Controls whether the client verifies server certificates and host name.
      # If insecure is set to true, all server host names and certificates will be
      # accepted. In this mode TLS based connections are susceptible to
      # man-in-the-middle attacks. Use only for testing.
      #insecure: true

      # Configure cipher suites to be used for TLS connections
      #cipher_suites: []

      # Configure curve types for ECDHE based cipher suites
      #curve_types: []

      # Configure minimum TLS version allowed for connection to logstash
      #min_version: 1.0

      # Configure maximum TLS version allowed for connection to logstash
      #max_version: 1.2


  ### Logstash as output
  #logstash:
    # The Logstash hosts
    #hosts: ["localhost:5044"]

    # Number of workers per Logstash host.
    #worker: 1

    # The maximum number of events to bulk into a single batch window. The
    # default is 2048.
    #bulk_max_size: 2048

    # Set gzip compression level.
    #compression_level: 3

    # Optional load balance the events between the Logstash hosts
    #loadbalance: true

    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: filebeat

    # Optional TLS. By default is off.
    #tls:
      # List of root certificates for HTTPS server verifications
      #certificate_authorities: ["/etc/pki/root/ca.pem"]

      # Certificate for TLS client authentication
      #certificate: "/etc/pki/client/cert.pem"

      # Client Certificate Key
      #certificate_key: "/etc/pki/client/cert.key"

      # Controls whether the client verifies server certificates and host name.
      # If insecure is set to true, all server host names and certificates will be
      # accepted. In this mode TLS based connections are susceptible to
      # man-in-the-middle attacks. Use only for testing.
      #insecure: true

      # Configure cipher suites to be used for TLS connections
      #cipher_suites: []

      # Configure curve types for ECDHE based cipher suites
      #curve_types: []


  ### File as output
  #file:
    # Path to the directory where to save the generated files. The option is mandatory.
    #path: "/tmp/filebeat"

    # Name of the generated files. The default is `filebeat` and it generates files: `filebeat`, `filebeat.1`, `filebeat.2`, etc.
    #filename: filebeat

    # Maximum size in kilobytes of each file. When this size is reached, the files are
    # rotated. The default value is 10 MB.
    #rotate_every_kb: 10000

    # Maximum number of files under path. When this number of files is reached, the
    # oldest file is deleted and the rest are shifted from last to first. The default
    # is 7 files.
    #number_of_files: 7


  ### Console output
  # console:
    # Pretty print json event
    #pretty: false


############################# Shipper #########################################

shipper:
  # The name of the shipper that publishes the network data. It can be used to group
  # all the transactions sent by a single shipper in the web interface.
  # If this options is not defined, the hostname is used.
  #name:

  # The tags of the shipper are included in their own field with each
  # transaction published. Tags make it easy to group servers by different
  # logical properties.
  #tags: ["service-X", "web-tier"]

  # Uncomment the following if you want to ignore transactions created
  # by the server on which the shipper is installed. This option is useful
  # to remove duplicates if shippers are installed on multiple servers.
  #ignore_outgoing: true

  # How often (in seconds) shippers are publishing their IPs to the topology map.
  # The default is 10 seconds.
  #refresh_topology_freq: 10

  # Expiration time (in seconds) of the IPs published by a shipper to the topology map.
  # All the IPs will be deleted afterwards. Note, that the value must be higher than
  # refresh_topology_freq. The default is 15 seconds.
  #topology_expire: 15

  # Internal queue size for single events in processing pipeline
  #queue_size: 1000

  # Configure local GeoIP database support.
  # If no paths are not configured geoip is disabled.
  #geoip:
    #paths:
    #  - "/usr/share/GeoIP/GeoLiteCity.dat"
    #  - "/usr/local/var/GeoIP/GeoLiteCity.dat"


############################# Logging #########################################

# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  #to_syslog: true

  # Write all logging output to files. Beats automatically rotate files if rotateeverybytes
  # limit is reached.
  #to_files: false

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    #path: /var/log/mybeat

    # The name of the files where the logs are written to.
    #name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7

  # Enable debug output for selected components. To enable all selectors use ["*"]
  # Other available selectors are beat, publish, service
  # Multiple selectors can be chained.
  #selectors: [ ]

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  #level: error

strings instead of int in the config

I set file output, but when it marshals out to the YAML, it's a string (even if I set it without quotes)

  rotate_every_kb: "1024"
  number_of_files: "3"

beats hates it:

Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: unmarshal errors:
line 12: cannot unmarshal !!str '1024' into int
line 13: cannot unmarshal !!str '3' into int. Exiting.

Issue downloading from puppet forge

Puppetserver 2.2.1
Amazon Linux

puppet module install pcfens-filebeat
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Error: Could not install module 'pcfens-filebeat' (???)
  No version of 'pcfens-filebeat' can satisfy all dependencies
    Use `puppet module install --ignore-dependencies` to install only this module
puppet module install pcfens-filebeat --ignore-dependencies
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└── pcfens-filebeat (v0.5.7)

filebeat5.yml template requires setting empty partition option for Kafka output

I'm in the process of deploying Filebeat v5 with Kafka output. I get an error if I do not provide a partition option to the Kafka configuration.

This is my current configuration:

class { '::filebeat':
  manage_repo => false,
  registry_file => '/var/lib/filebeat/registry',
  outputs => {
    'kafka' => {
      'enabled' => true,
      'hosts' => $kafka_hosts,
      'topic' => '%{[type]}',
      'compression' => 'gzip',
      'ssl' => {
        'enabled' => true,
      }
    },
  },
}

With this configuration, however, I receive the following compilation failure:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template filebeat/filebeat5.yml.erb:
  Filepath: /etc/puppet/environments/test/module_repos/filebeat/templates/filebeat5.yml.erb
  Line: 276
  Detail: undefined method `[]' for nil:NilClass
 at /etc/puppet/environments/test/module_repos/filebeat/manifests/config.pp:29 on node test
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

The following configuration is necessary to get the template to compile:

class { '::filebeat':
  manage_repo => false,
  registry_file => '/var/lib/filebeat/registry',
  outputs => {
    'kafka' => {
      'enabled' => true,
      'hosts' => $kafka_hosts,
      'topic' => '%{[type]}',
      'compression' => 'gzip',
      'partition' => {},
      'ssl' => {
        'enabled' => true,
      }
    },
  },
}

Having a dangling empty config option is rather non-elegant however. Can this be fixed such that I can remove the partition element?

Typo in logstash/elasticsearch TLS configuration

Howdy y'all!

Found a quick typo in the filebeat.yaml templates for logstash and elasticsearch's TLS configuration:

https://github.com/pcfens/puppet-filebeat/blob/master/templates/filebeat.yml.ruby18.erb#L81-L83
https://github.com/pcfens/puppet-filebeat/blob/master/templates/filebeat.yml.ruby18.erb#L137-L139

Looks like you're checking for certifcate_key (typo, missing i in certificate) and then using certificate_key (spelled correctly) in the actual output. 😄

Variable scoping

the ERB templates have variables that do not pull from the correct scope.

filebeat:
spool_size: <%= @spool_size %>
idle_timeout: <%= @idle_timeout %>
registry_file: <%= @registry_file %>
config_dir: <%= @config_dir %>

This results in a yml file without values. This needs to change to

filebeat:
spool_size: <%= scope['::filebeat::spool_size'] %>
idle_timeout: <%= scope['::filebeat::idle_timeout'] %>
registry_file: <%= scope['::filebeat::registry_file'] %>
config_dir: <%= scope['::filebeat::config_dir'] %>

I'm looking into changing this. will submit a PR when done

Wiping the defaults of filebeats

The current implementation of the filebeat.yml template, wipes all the default filebeat settings. It would be better to make sure that if the variable (such as logging), is not passed to the class that it isn't set in the yaml file.

Incorrect prospector template

Hello and thanks for your great module!

I tried to use "include_lines" without succes because the prospector template defines it as

  include_lines:
    - []
    - []

When I switched the syntax to the one from the official documentation it filtered the log accordingly.
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

The following example configures Filebeat to export any lines that start with "ERR" or "WARN":

include_lines: ["^ERR", "^WARN"]

Ca you please check it out.

Thanks!

Add Multiple instances support

Hi @pcfens,
We ship most of our logs to Logstash for parsing and enrichment.
As some of our logs are already formatted as JSON objects and do not need any modification, we can ship those lines directly to ES.

The dependencies for this feature are:

  1. Install FileBeat 5.x (which introduced JSON parsing feature, among other features)
  2. Support installing multiple instances of FileBeat on the same host.

The later is due because FileBeat itself doesn't have the ability to configure which prospector ships through which output.

You're more then welcome to take a look on this fork. the above bullets are implemented and working.

I'd be happy to get your take on the above :)

N.B.
I still need to modify the tests to the new structure and solve some Docker issues preventing Beaker tests from running.

Thank you,
Yarden

Assign a puppet manifest to a group host

**Question

I have a puppet running with a Foreman node to manage different hosts.

I installed and loaded puppet-filebeat with success and have access to class parameters in my Foreman web interface. I can assign different values to my parameters and assign the module to a host group to test and eventually to production host group to deploy.

The problem is I want to manage my SSL certificate so I created a manifest that looks like the one in the Wiki and created a mount point on my puppet server to gather the certs.

I don't know how to assign my puppet manifest to a Foreman hostgroup. I am not the person that installed the puppet server / foreman node so I am quite helpless with the product.

If you have any suggestion please let me know.
Thank you for your time

Missing "port" option for logstash config

As mentioned in the filebeat docs, there's a port option so you don't have to specify the port for each host in hosts. This is useful if you want to, for example, query your list of logstash hosts from PuppetDB, which will result in raw hostnames without ports suffixed.

Incorrect indentation and/or mixed format in filebeat.yml causing service to not start

Filebeat.yml extract:
#================================ Outputs =====================================

#----------------------------- Logstash output --------------------------------
output.logstash:
hosts:
- logstash.********************

enabled: true
certificate_authorities:
  - /var/lib/*********************
certificate: "/var/lib/***************************"
key: "/var/lib/*********************************"

Error message:

service filebeat start

Starting filebeat: Exiting: error loading config file: yaml: line 59: did not find expected '-' indicator

expanding output.logstash to
output:
logstash:

fixes the problem as does removing the two leading spaces on the enabled, certifcate_authorities, certificate, and key.

logging Variable order on puppet 3.x with Ruby 1.8

All Variable vaule has be changed , and will not affect the order?

Test steps are as follows

  1. define node.pp
node 'hostname' {
    logging = > {
        'level' = > info,
        'to_files' = > false,
        'to_syslog' = > false,
        'files' = > {
            path = > '/var/log/filebeat',
            keepfiles = > '7',
            name = > 'filebeat.log',
        },
    },
}
  1. Corresponding to the generated files
# more /etc/filebeat/filebeat.yml
……
  logging:
    to_syslog: false
    to_files: false
    level: info
    files:
      path: /var/log/filebeat
      name: filebeat.log
      keepfiles: 7
  1. Change file order
node 'hostname' {
    logging = > {
        'files' = > {
            path = > '/var/log/filebeat',
            keepfiles = > '7',
            name = > 'filebeat.log',
        },
        'level' = > info,
        'to_files' = > false,
        'to_syslog' = > false,
    },
}
  1. Didn't change anything with puppet
  2. because no value change*
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for monster-agent-test-172-16-14-237
Info: Applying configuration version '1460095419'
Notice: Finished catalog run in 0.19 seconds
  1. Change to_files value , Puppet Changed, but order has not changed.
node 'hostname' {
    logging = > {
        'files' = > {
            path = > '/var/log/filebeat',
            keepfiles = > '7',
            name = > 'filebeat.log',
        },
        'level' = > info,
        'to_files' = > false,
        'to_syslog' = > false,
    },
}
+++ /tmp/puppet-file20160408-25677-10ar15d-0    2016-04-08 14:08:19.505687725 +0800
@@ -19,7 +19,7 @@

   logging:
     to_syslog: false
-    to_files: false
+    to_files: true

     files:
       path: /var/log/filebeat

Wish you a good day, and i'm looking forward to your reply.

Add Winlogbeat support

Filebeat support only log files, but for windows eventlog there is winlogbeat program. Please add support for it in your module.

puppet-filebeat 0.8.0 breaks backward compatibility for filebeat 1.x

Pre-0.8.0 the module always assumes a major version of 1 (not really a parameter, but the manifest always creates a 1.x compatible configfile).

In 0.8.0 the module assumes $major_version of 5 if own fact $::filebeat_version can not be found. However $::filebeat_version needs a installed filebeat client to work, but this is installed as part of the inclusion. This leads to the error when updating the module (minor version update) without addin parameters:

  • 1st run:

    • fact $::filebeat_version is empty as no package is installed yet
    • run filebeat module which installs filebeat package (from our mirror (which is 1.x)) and a filebeat 5.0 config file (major_version of 5 assumed)
    • start of filebeat client failes (config file incompatible output.logstash ...)
  • 2nd run:

    • fact $::filebeat_version is 1.x as package is installed
    • run filebeat which installs and a filebeat 1.0 config file
    • start of filebeat client ok

Fix:
Assume default of major_version 1 to maintain backward compatibility.

filebeat unable to parse yaml for processors created by sensu-puppet

with the latest puppet-filebeat (that installs filebeat 5.x)
commit: adc8045

   class { 'filebeat':
     outputs => {
       'logstash' => {
         'bulk_max_size' => 1024,
         'hosts' => [
           'localhost:5044',
         ],
       },
       'file'     => {
         'path' => '/tmp',
         'filename' => 'filebeat',
         'rotate_every_kb' => 10240,
         'number_of_files' => 2,
       },
     },
     shipper => {
       refresh_topology_freq => 10,
       topology_expire => 15,
       queue_size => 1000,
     },
     logging => {
       to_files => true,
       level => 'debug',
       files => {
         rotateeverybytes => 10485760,
         keepfiles => 7,
       }
     },
     prospectors => {
       'sensu-events-to-logstash' => {
         doc_type => 'json',
         paths    => [
           '/var/log/sensu/sensu-client.log',
         ],
         fields   => {
           payload => 'payload',
           check   => 'check',
         },
         json => {
           message_key     => "message",
           keys_under_root => true,
           add_error_key   => true
         },
         tags     => [
           'tag1',
           'tag2',
           'tag3',
         ]
       }
     },
     processors => [
       { 'name' => 'drop_event', 'when' => { 'or' => [{'equals' => {'http.code' => 200}}, {'equals' => {'http.code' => 300}}] } }
     ]
   }

## puppet agent -t 
Notice: /Stage[main]/Filebeat::Config/File[filebeat.yml]/content:
--- /etc/filebeat/filebeat.yml	2017-01-23 18:26:48.813100221 +0000
+++ /tmp/puppet-file20170124-9115-1o6lngu	2017-01-24 06:29:24.090281278 +0000
@@ -50,8 +50,14 @@
 #           http.code: 200
 #
 processors:
-  - include_fields:
-      fields: ["cpu"]
+  - drop_event:
+      when:
+        ---    ####THIS MAKES THE YAML INVALID?
+          or:
+            - equals:
+                http.code: "200"
+            - equals:
+                http.code: "300"

 #================================ Outputs =====================================


Info: Computing checksum on file /etc/filebeat/filebeat.yml
Info: FileBucket got a duplicate file {md5}4a63a6d27ce4272e1e407aaa6d662d09
Info: /Stage[main]/Filebeat::Config/File[filebeat.yml]: Filebucketed /etc/filebeat/filebeat.yml to puppet with sum 4a63a6d27ce4272e1e407aaa6d662d09
Debug: Executing '/usr/share/filebeat/bin/filebeat -N -configtest -c /etc/filebeat/filebeat.yml20170124-9115-roa5zd'
Error: Execution of '/usr/share/filebeat/bin/filebeat -N -configtest -c /etc/filebeat/filebeat.yml20170124-9115-roa5zd' returned 1: Exiting: error loading config file: yaml: line 55: mapping values are not allowed in this context
Error: /Stage[main]/Filebeat::Config/File[filebeat.yml]/content: change from {md5}4a63a6d27ce4272e1e407aaa6d662d09 to {md5}14888eb836dc71b293df5a107d9d0c75 failed: Execution of '/usr/share/filebeat/bin/filebeat -N -configtest -c /etc/filebeat/filebeat.yml20170124-9115-roa5zd' returned 1: Exiting: error loading config file: yaml: line 55: mapping values are not allowed in this context
Notice: /Stage[main]/Filebeat::Service/Service[filebeat]: Dependency File[filebeat.yml] has failures: true
Warning: /Stage[main]/Filebeat::Service/Service[filebeat]: Skipping because of failed dependencies
Notice: /Stage[main]/Filebeat/Anchor[filebeat::end]: Dependency File[filebeat.yml] has failures: true
Warning: /Stage[main]/Filebeat/Anchor[filebeat::end]: Skipping because of failed dependencies

also the yaml cannot be parsed by std languages also

root@ubuntu1404-2:~# cat /tmp/foo
processors:
  - include_fields:
      fields: ["cpu"]
  - drop_event:
      when:
        ---
          or:
            - equals:
                http.code: "200"
            - equals:
                http.code: "300"
root@ubuntu1404-2:~# ruby -le 'require "yaml"; p YAML.load(File.read("/tmp/foo"))'
/usr/lib/ruby/2.0.0/psych.rb:205:in `parse': (<unknown>): mapping values are not allowed in this context at line 7 column 13 (Psych::SyntaxError)
	from /usr/lib/ruby/2.0.0/psych.rb:205:in `parse_stream'
	from /usr/lib/ruby/2.0.0/psych.rb:153:in `parse'
	from /usr/lib/ruby/2.0.0/psych.rb:129:in `load'
	from -e:1:in `<main>'

"Duplicate declaration: File[undef] is already declared" on Windows system

Receiving the following message when updating the agent on a Windows 2008 R2 system:

Duplicate declaration: File[undef] is already declared in file /etc/puppet/environments/env/modules/filebeat/manifests/install/windows.pp:7; cannot redeclare at /etc/puppet/environments/env/modules/filebeat/manifests/install/windows.pp:11

windows.pp contents:
`class filebeat::install::windows {
$filename = regsubst($filebeat::download_url, '^https./([^\/]+).[^.].', '\1')
$foldername = 'Filebeat'

file { $filebeat::tmp_dir:
ensure => directory
}

file { $filebeat::install_dir:
ensure => directory
}

remote_file {"${filebeat::tmp_dir}/${filename}.zip":
ensure => present,
source => $filebeat::download_url,
require => File[$filebeat::tmp_dir],
verify_peer => false,
}

exec { "unzip ${filename}":
command => "$sh=New-Object -COM Shell.Application;$sh.namespace((Convert-Path '${filebeat::install_dir}')).Copyhere($sh.namespace((Convert-Path '${filebeat::tmp_dir}/${filename}.zip')).items(), 16)",
creates => "${filebeat::install_dir}/Filebeat",
provider => powershell,
require => [
File[$filebeat::install_dir],
Remote_file["${filebeat::tmp_dir}/${filename}.zip"],
],
}

exec { 'rename folder':
command => "Rename-Item '${filebeat::install_dir}/${filename}' Filebeat",
creates => "${filebeat::install_dir}/Filebeat",
provider => powershell,
require => Exec["unzip ${filename}"],
}

exec { "install ${filename}":
cwd => "${filebeat::install_dir}/Filebeat",
command => './install-service-filebeat.ps1',
onlyif => 'if(Get-WmiObject -Class Win32_Service -Filter "Name='filebeat'") { exit 1 } else {exit 0 }',
provider => powershell,
require => Exec['rename folder'],
}
}
`

Adding topbeat and packetbeat to the same module

Hi there

First of all, thanks for the great work on this module. After trying the other modules which are out there for making filebeat work i found yours and it works splendid!

Because there's not just filebeat. I was thinking if it would make sense to have topbeat and packetbeat also in the same puppet module, because the configuration files are pretty similar and the repository where we install from is also the same.

What do you think about it?

Best,
/dasrecht

cant get doc_type to apply

using foreman for my ENC with data defined as such, i cant get the doc_type to apply.

root@phxlv-log01:/usr/share/filebeat/bin# ./filebeat --version
filebeat version 5.0.1 (amd64), libbeat 5.0.1

filebeat::outputs (hash)

logstash:
  index: filebeat
  hosts:
  - phxlv-log01.vm.slan.devita.co:5044

filebeat::prospectors (hash)

syslogs:
  doc_type: syslog
  paths:
  - "/var/log/syslog"
authlogs:
  doc_type: authlog
  paths:
  - "/var/log/auth.log"

the file in /etc/filebeats/conf.d/authlogs.yml looks like this

---
filebeat:
  prospectors:
    - input_type: log
      paths:
        - /var/log/auth.log
      encoding: plain
      fields_under_root: false
      scan_frequency: 10s
      harvester_buffer_size: 16384
      max_bytes: 10485760

plugin broken, typo

Detail: Could not find value for 'vvalue' at /etc/puppet/environments/production/modules/filebeat/templates/prospector.yml.erb:19

change
<%= key %>: <%= vvalue %>
to
<%= key %>: <%= value %>

to_yaml changes everytime

Hello, I install filebeat.

Although generating facts.yaml dynamically is definitely something i want ,but Puppet will change it almost everytime .

I search one-blog to fix it . I try to change <%= @filebeat_config.to_yaml() %> to <%= @filebeat_config.to_yaml.sort() %> and any parameters is did't work .

some many people encounter this problem. say: "I have the same problem and this is not a bug in the module itself. In the routes.yaml.erb template there's a call to the method 'to_yaml', which in Ruby 1.8.7 is not stable which means the order of the dumped lines is not always the same. In Ruby >=1.9 this is no longer the case.
Unless the module developer's do not want the replace the call to_yaml with something dirty and very hacky for us Ruby 1.8.7 users, we will have to live with this, I'm afraid."

My env :

# ruby --version
ruby 1.8.7 (2012-02-08 patchlevel 358) [x86_64-linux]

# puppet --version
3.7.3

Default settings do not work on Windows

There's space in default path C:\Program Files\Filebeat so this will fail in prospectors.pp:

 validate_cmd => "${filebeat_path} -N -configtest -c %",

Windows doesn't understand spaces in path. It should probably be:

 validate_cmd => "\"${filebeat_path}\" -N -configtest -c \"%\"",

no support for output.console

Defining console output in Puppet via:

  outputs:
    console:
      pretty: true

has no effect in the resultant Filebeat config file, because templates/filebeat5.yml.erb has no section for output.console. Can you add it, for completeness sake? Console output is useful at the trial-and-error stage to get your output format right, before passing it to Kafka or else.

Puppet Forge reports that the module doesn't validate on Puppet 3.8

I can't re-create the issue on my 3.8 VM with the puppet-syntax gem or the puppet parser validate command. I've sent a question to puppet, but wanted to get something here in case there are questions/discussion.

I was able to apply a test manifest to my Puppet 3.8 vagrant box with the latest version, so hopefully there won't be any real issues.

class instantiation failing

Hi,
I downloaded the module for filebeat and tried the default class declaration as below:

node 'fbes1.hs.com'{
include filebeat
class { 'filebeat':
outputs => {
'logstash' => {
'hosts' => ['lsredis.hs.com:5044',],
'loadbalance' => true,
},
},
}
}

but the class declaration fails with below error:

[root@fbes1 ~]# puppet agent -t --noop
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: compile error
/etc/puppet/modules/filebeat/templates/filebeat3.yml.erb:6: syntax error, unexpected ')'
) ).to_s)
^
/etc/puppet/modules/filebeat/templates/filebeat3.yml.erb:7: syntax error, unexpected tIDENTIFIER, expecting ')'
_erbout
^
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

maybe I'm doing something wrong. Please let me know if this needs any fixing.

filebeat::config called when no config_template set

So using manifest like this:

class {'filebeat':
  download_url => "https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.1.1-windows-x86_64.zip",
}

Returns the following error with validate_cmd:
Trying to update the filebeats.yml with contents of template filebeat5.yml.erb.

Error: Execution of '"c:\Program Files\Filebeat\filebeat.exe" -N -configtest -c "C:/Program Files/Filebeat/filebeat.yml20170125-2496-86inoa"' returned 1: 2017/01/25 22:07:56 No paths given. What files do you want me to watch?
Error: /Stage[main]/Filebeat::Config/File[filebeat.yml]/content: change from {md5}73792f0b07e3e1bfc10007175bf43a1d to {md5}83f39c18a09cd309b1949c6e9ab1bf7f failed: Execution of '"c:\Program Files\Filebeat\filebeat.exe" -N -configtest -c "C:/Program Files/Filebeat/filebeat.yml20170125-2496-86inoa"' returned 1: 2017/01/25 22:07:56 No paths given. What files do you want me to watch?
Notice: /Stage[main]/Filebeat::Service/Service[filebeat]: Dependency File[filebeat.yml] has failures: true
Warning: /Stage[main]/Filebeat::Service/Service[filebeat]: Skipping because of failed dependencies
Notice: /Stage[main]/Filebeat/Anchor[filebeat::end]: Dependency File[filebeat.yml] has failures: true
Warning: /Stage[main]/Filebeat/Anchor[filebeat::end]: Skipping because of failed dependencies

Guessing we need a Windows default config to ship with?

Clarify license

The metadata file suggests this is an Apache 2.0 license, could this be clarified, an appropriate license file added, and copyright notice included if that is the intended distribution license?

Thanks

Package Update Does Not Notify Service

On a recent update to Package["filebeat"] I notice that messages were still being logged under the previous version. I upgraded from 5.0.0 to 5.0.1.

The logs state that Package["filebeat"] was updated and the next entry says "Finished catalog run"

Requesting that Class["filebeat::install", "filebeat::install::*"] notify Class["filebeat::service"] when the Package is updated.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.