createJSON function does not prevent base64 encoding

fluent-bit redis output plugin

This plugin is used to have redis output from fluent-bit. You can use fluent-bit redis instead of logstash in a configuration where you have a redis and optional stunnel in front of your elasticsearch infrastructure.

The configuration typically looks like:

fluent-bit --> stunnel --> redis <-- logstash --> elasticsearch

If you have multiple elastic search servers, each covered with a redis cache in front it might look like this:

           /-> stunnel --> redis <-- logstash --> elasticsearch 
           |
fluent-bit --> stunnel --> redis <-- logstash --> elasticsearch
           |
           \-> stunnel --> redis <-- logstash --> elasticsearch

Usage

docker run -it --rm -v /path/to/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf majst01/fluent-bit-go-redis-output

Building

docker build --no-cache --tag fluent-bit-go-redis-output .
docker run -it --rm -v /path/to/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf fluent-bit-go-redis-output

Configuration Options

Key	Description	Default
Hosts	Host(s) of redis servers, whitespace separated ip/host:port	127.0.0.1:6379
Password	Optional redis password for all redis instances	""
DB	redis database (integer)	0
UseTLS	connect to redis with tls	False
TlsSkipVerify	if tls is configured skip tls certificate validation for self signed certificates	True
Key	the key where to store the entries in redis	"logstash"

Example:

add this section to fluent-bit.conf

[Output]
    Name redis
    Match *
    UseTLS true
    TLSSkipVerify true
    # if port is ommited, 6379 is used
    Hosts 172.17.0.1 172.17.0.1:6380 172.17.0.1:6381 172.17.0.1:6382 172.17.0.1:6383
    Password homer
    DB 0
    Key elastic-logstash

Useful links

Redis format

logrus-redis-hook

Logstash Redis Output

logstash-redis-docu

TODO

Strategies for redis connection error handling

crash on connection errors

Given a list of 4 Redis databases, we pick on start a random one, if during operation this fails we panic and on restart the next hopefully working is selected.

rely on FLB_RETRY

With a list of redis databases we can create a list of pools, one pool per database and instead of doing a pool.Get(), call list.Get() with selects the next random redis database. If a failure occurs return FLB_RETRY and the library will retry.

	func createJSON(timestamp time.Time, tag string, record map[interface{}]interface{}) (*logmessage, error) {
	m := make(map[string]interface{})
	// convert timestamp to RFC3339Nano which is logstash format
	m["@timestamp"] = timestamp.UTC().Format(time.RFC3339Nano)
	m["@tag"] = tag
	for k, v := range record {
	switch t := v.(type) {
	case []byte:
	// prevent encoding to base64
	m[k.(string)] = string(t)
	default:
	m[k.(string)] = v
	}
	}
	js, err := json.Marshal(m)
	if err != nil {
	return nil, fmt.Errorf("error creating message for REDIS: %v", err)
	}
	return &logmessage{data: js}, nil
	}

majst01 / fluent-bit-go-redis-output Goto Github PK

fluent-bit-go-redis-output's Introduction

fluent-bit redis output plugin

Usage

Building

Configuration Options

Useful links

Redis format

Logstash Redis Output

TODO

Strategies for redis connection error handling

fluent-bit-go-redis-output's People

Contributors

Stargazers

Watchers

Forkers

fluent-bit-go-redis-output's Issues

Recommend Projects

Recommend Topics

Recommend Org