Manage Fluentd installation, configuration and Plugin-management with Puppet using the td-agent.
- Debian (tested on Debian 7.5)
- Ubuntu
- Redhat
- CentOS (tested on CentOS 6.4)
- apt: "https://github.com/puppetlabs/puppetlabs-apt.git" (Only for Debian)
- stdlib: "https://github.com/puppetlabs/puppetlabs-stdlib.git"
- Fork it
- Create a feature branch (
git checkout -b my-new-feature) - Run rspec tests (
bundle exec rake specorrake spec) - Commit your changes (
git commit -am 'Added some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
- master tested and working. Latest Version on http://forge.puppetlabs.com
- developmemt includes the newewst features. works too, mostly.
How to configure a Agent to send data to a centralised Fluentd-Server
Install your fluentd plugin. (Check here for the right plugin name.)
You can choose from a file or gem based installation.
include ::fluentd
fluentd::install_plugin { 'elasticsearch':
plugin_type => 'gem',
plugin_name => 'fluent-plugin-elasticsearch',
}
The agent watches over your logfiles and sends its content to the collector.
include ::fluentd
fluentd::source { 'apache':
config => {
'format' => 'apache2',
'path' => '/var/log/apache2/access.log',
'pos_file' => '/var/tmp/fluentd.pos',
'tag' => 'apache.access_log',
'type' => 'tail',
},
}
fluentd::source { 'syslog':
config => {
'format' => 'syslog',
'path' => '/var/log/syslog',
'pos_file' => '/tmp/td-agent.syslog.pos',
'tag' => 'system.syslog',
'type' => 'tail',
},
}
fluentd::match { 'forward':
pattern => '**',
priority => '80',
config => {
'type' => 'forward',
'servers' => [
{ 'host' => 'fluentd.example.com', 'port' => '24224' }
],
},
}
# ensure an old config is no longer there
fluentd::source { 'some_source':
ensure => 'absent',
}
...creates the following files:
/etc/td-agent/
├── config.d
│ ├── 50-source-apache.conf
│ ├── 50-source-syslog.conf
│ └── 80-match-forward.conf
├── ...
...
The Collector collects all the data from the Agents. He now stores the data in files, Elasticsearch or elsewhere.
include ::fluentd
fluentd::source { 'collector':
priority => '10',
config => {
'type' => 'forward',
}
}
fluentd::match { 'collector':
pattern => '**',
config => {
'type' => 'elasticsearch',
'logstash_format' => true,
},
}
# all rsyslog daemons on the clients sends their messages to 5140
fluentd::source { 'rsyslog':
type => 'syslog',
config => {
'port' => '5140',
'bind' => '0.0.0.0',
'tag' => 'system.local',
},
}
...creates the following files:
/etc/td-agent/
├── config.d
│ ├── 10-source-collector.conf
│ ├── 50-match-collector.conf
│ └── 50-source-rsyslog.conf
├── ...
...
An array of configurations implies type "copy".
$logger=[ { 'host' => 'logger-sample01', 'port' => '24224'},
{ 'host' => 'logger-example01', 'port' => '24224', 'standby' => ''} ]
fluentd::match { 'forward_to_logger':
pattern => 'alocal',
config => [
{
'type' => 'forward',
'send_timeout' => '60s',
'recover_wait' => '10s',
'heartbeat_interval' => '1s',
'phi_threshold' => 8,
'hard_timeout' => '60s',
'flush_interval' => '5s',
'servers' => $logger,
},
{
'type' => 'stdout',
'output_type' => 'json',
}
],
}
```
### add a filter
```
fluentd::filter { 'myfilter':
pattern => '**',
config => {
'type' => 'grep',
'input_key' => 'key',
'regexp' => '/*.foo.*/',
'exclude' => 'baar',
'output_tag' => 'mytag',
'add_tag_prefix' => 'pre_',
'remove_tag_prefix' => 'remove_',
'add_tag_suffix' => '_after',
'remove_tag_suffix' => '_remove',
'customvalue' => true,
}
}
```
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent