bshuster-repo / logrus-logstash-hook Goto Github PK

View Code? Open in Web Editor NEW

196.0 11.0 81.0 58 KB

:information_source: Logstash hook for logrus

License: MIT License

Go 100.00%

logstash logrus hook golang hacktoberfest

logrus-logstash-hook's Introduction

Logstash hook for logrus

Use this hook to send the logs to Logstash.

Usage

package main

import (
        "github.com/bshuster-repo/logrus-logstash-hook"
        "github.com/sirupsen/logrus"
        "net"
)

func main() {
        log := logrus.New()
        conn, err := net.Dial("tcp", "logstash.mycompany.net:8911")
        if err != nil {
                log.Fatal(err)
        }
        hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "myappName"}))

        log.Hooks.Add(hook)
        ctx := log.WithFields(logrus.Fields{
                "method": "main",
        })
        ctx.Info("Hello World!")
}

This is how it will look like:

{
    "@timestamp" => "2016-02-29T16:57:23.000Z",
      "@version" => "1",
         "level" => "info",
       "message" => "Hello World!",
        "method" => "main",
          "host" => "172.17.0.1",
          "port" => 45199,
          "type" => "myappName"
}

FAQ

Q: I would like to add characters to each line before sending to Logstash? A: Logrustash gives you the ability to mutate the message before sending it to Logstash. Just follow this example.

Q: Is there a way to maintain the connection when it drops A: It's recommended to use GoAutoSocket for that. See here how it can be done.

Maintainers

Name	Github
Boaz Shuster	boaz0

License

MIT.

logrus-logstash-hook's People

Contributors

Stargazers

Watchers

logrus-logstash-hook's Issues

issues with "go modules"

Hi,

I've tried to convert my project to go modules.
By default the last release 0.41.0is chosen which is from 2017 and quite older than master.

I think it would be wise to create a new release from time to time.

Current workaround:

Update you go.mod file to use the master branch instead of version.

Not Working with dep

Master Code and Release source code has different folder stucture

Works with standard go get, but not works with dep

How to change timestamp format from string to Unix format float64 or int64 ??
by default it gives "@timestamp" => "2016-02-29T16:57:23.000Z",
I want to pass time as Unix time "@timestamp" => 1611664414.310306.
And if it possible to leave time as is in time.RFC3339 format.
e.g. outgoing message should be like this

{
  "@level": "info",
  "@message": "Init logger with Logstash!",
  "time": "2021-01-26T19:10:59+03:00",
  "@timestamp": 1611677459.123
}

The only thing I succeed in is that I can add such timestamp in extra fields.

hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"hostname": "my_sginy_host", "@timestamp":time.Now().Unix()}))
logger.Hooks.Add(hook)

Such code adds @timestamp in the fields section.

{
  "@timestamp": "2021-01-26T19:10:59+03:00",
  "@version":"1",
  "fields.@timestamp": 1611677459,
  "hostname": "my_shiny_host",
  "@level": "info",
  "message": "Init.redis!", "type":"log"
}

v0.4 is not working anymore with logrus v1.0.3

This is an informal issue.

Since the v1.0.3 or loggrus, the commit 325575f18110626a8fd7f027f899aa8c8f5169c6 makes the v0.4 to fail because of the dependency of DefaultTimestampFormat logstash_formatter.go

Note that the v1.0-beta is not impacted by this issue.

Reconnect to logstash on connection timeout

After long stretches of inactivity our logstash connection times out with the following errors:

Failed to fire hook: write tcp 10.0.0.212:45992->10.0.4.98:5000: write: connection timed out
Failed to fire hook: write tcp 10.0.0.212:45992->10.0.4.98:5000: write: broken pipe

As such, logs are no longer written. Is there any way to have the logstash hook automatically retry the connection?

Reconnect on connection loss

Sometimes (server restart, intermittent network outages) a TCP connection breaks, and is never restored unless the app is restarted or a hook is recreated anew.

I'd propose implementing automatic reconnections when possible (i.e. persist protocol and address in Hook if they were provided). Easiest solution would be to just re-Dial if h.conn.Write fails, but since logging is synchronous and starting a TCP connection when the network is down may take a whole minute, it may essentially lock up the caller.

If desirable, I can work on a pull request that'd move network-sending code into a separate goroutine, which would receive data from a buffered channel (or, if introducing a new third-party dependency is acceptable, a RingChannel).

What do you think?

case-insensitive import collision

The uppercase "Sirupsen" in this project's import path collides with the lower-case variant.

When trying to compile on macOS this error occurs:

case-insensitive import collision: "github.com/Sirupsen/logrus" and "github.com/sirupsen/logrus"

I can not switch to an upper-case import path because other hooks I use, use the lower-case variant:

cannot use logger (type *"github.com/Sirupsen/logrus".Logger) as type *"github.com/sirupsen/logrus".Logger in assignment

No caller report

I try to set Report caller
Logger.SetReportCaller(true)

but caller report not send to logstash

@version should be a string

Just something I noticed in your readme
The @Version field should really be a string:
https://github.com/logstash-plugins/logstash-codec-json/blob/master/spec/codecs/json_spec.rb#L26

New API for logrus-logstash-hook

In the next version (1.xx) I am going to do a major change around the API:

Deprecate the current logstash formatter in favor of Logrus' JSON Format.
WithPrefix, WithFields and WithField, NewHookWithConn and etc will be removed.
The Hook structure will be changed.
Easy way to format and to integrate with Go packages

README on master out of date

This was a bit annoying but I found that the README that displays on github is out of date, the README found in vendor/github/bshuster-repo/logrus-logstash-hook is up to date however.

Its not wholly intuitive for someone to check the vendor files for help on installed packages.

Example from readme doesn't compile for me

package main

import (
	"github.com/Sirupsen/logrus"
	"github.com/bshuster-repo/logrus-logstash-hook"
)

func main() {
	log := logrus.New()
	hook, err := logrus_logstash.NewHook("tcp", "172.17.0.2:9999", "myappName")

	if err != nil {
		log.Fatal(err)
	}

	log.Hooks.Add(hook)

	ctx := log.WithFields(logrus.Fields{
		"method": "main",
	})

	ctx.Info("Hello World!")
}

errors with

# command-line-arguments
./logstash.go:16: cannot use hook (type *logrus_logstash.Hook) as type "..../vendor/github.com/Sirupsen/logrus".Hook in argument to log.Hooks.Add:
	*logrus_logstash.Hook does not implement "..../vendor/github.com/Sirupsen/logrus".Hook (wrong type for Fire method)
		have Fire(*"github.com/Sirupsen/logrus".Entry) error
		want Fire(*"..../vendor/github.com/Sirupsen/logrus".Entry) error

What could be the problem?

Test error with Go 1.14beta1

Fedora Rawhide, with Go 1.14beta1, logrus-logstash-hook 0.4.1

Testing    in: /builddir/build/BUILD/logrus-logstash-hook-0.4.1/_build/src
         PATH: /builddir/build/BUILD/logrus-logstash-hook-0.4.1/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
       GOPATH: /builddir/build/BUILD/logrus-logstash-hook-0.4.1/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags "-X github.com/bshuster-repo/logrus-logstash-hook/version=0.4.1 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '"
      testing: github.com/bshuster-repo/logrus-logstash-hook
github.com/bshuster-repo/logrus-logstash-hook
--- FAIL: TestLogstashFormatter (0.00s)
    logstash_formatter_test.go:58: expected data[error] to be 'Get http://example.com: The error' but got 'Get "http://example.com": The error'
FAIL

Seem like the formatting changed slightly to include quotes.

what kind of `input` should I config at the logstash side?

I am trying making this logrus -> logstash pipeline work. I was using filebeats as my logstash's input. But what should I use for logrus? A simple logstash pipeline config sample is as follows.

input {
  # is there a "logrus" input for logrus?
  beats {
    port => 5044
  }
}

#filter {
# should I do any filtering stuff here?
#}


output {
  elasticsearch {
    hosts => "192.168.0.1:9200"
    manage_template => false
    index => "logstash-gateway-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

thanks a lot!

add microsecond to @timestamp plz.

The @timestamp did not have microsecond. It's 2017-04-24 18:54:18.000.

I think microsecond is important like 2017-04-24 18:54:18.231

 timeStampFormat = logrus.DefaultTimestampFormat
to
  timeStampFormat = time.RFC3339Nano

confusing versioning: 1.0 or 0.5 or 5.0?

Why the single commit b3d898b is tagged as 1.0, 0.5-beta, v5.0, v1.0 and v0.5 at the same time??

This is very confusing.

logstash configurationo filters

I think is good to create a wiki to tell how to configure the logstash config files according to your filters and the current format.

Doesn't work on UDP for big messages

It's just a topic for a discussion.

If I try to send a large message then it doesn't fit to MTU. This is from tcpdump:

10:35:03.216961 IP XXX.XXX.XXX.XXX.55505 > XXX.XXX.XXX.XXX.3535: UDP, bad length 1914 > 1472

However it doesn't seems to be the problem of this project. It seems to be a problem of used net.Conn… Or UDP just cannot be used for large messages in logstash at all.

So the question: what is the simplest way (if exists) to use this plugin over UDP?

Logstash reconnection handeling

After creating the connection with Dial, how the hook will be able to maintain the connection if it drops?

How it will reconnect?

Timestamp accuracy

Hi there,

I've set up this hook, and the timestamps are at whole second intervals, so sorting on the timestamps where there are more than 1 per second is somewhat broken.

for the console logs that come out of logrus I was able to get accurate timestamps like this:

log := logrus.New()
	fmt := new(logrus.TextFormatter)
	fmt.TimestampFormat = time.RFC3339Nano
	fmt.FullTimestamp = true
	log.SetFormatter(fmt)

for this hook though I'm unable to figure out how to get more accurate timestamps
This is the console:

This is kibana, sorted by timestamp (out of order):

Is there a way around this?

nothing written to logstash!

Hello,

I'm testing your library just like your example but nothing written in LogStash/Elasticsearch until I add a one second time delay before the program ended!
here the code it is:

func main() {
	log := logrus.New()
	conn, err := net.Dial("tcp", "127.0.0.1:8911")
	if err != nil || conn == nil {
		log.Fatal(err)
	}
	hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "myappName"}))

	log.Hooks.Add(hook)
	ctx := log.WithFields(logrus.Fields{
		"method": "main",
	})
	ctx.Info("Hello World in print again!")

	time.Sleep(1 * time.Second)
}

I'll appreciate if anybody can help

set log level

how can i set log level?
for example just error and fatal?

New release needed

Hi,
We need a new release to fix the bug with persistent fields added with WithField method

Thanks! :)

Hook permanently saves all fileds

Hi,

Hook and logrus can write logs with different set of fields.
For exmaple,

log.WithField("animal", "walrus"}).Info("First log")
log.Info("Second log")

Logrus output writes second log without animal field while hook will send log with animal equal to "walrus"

After inspecting hook's code I have found reason for this behavior. copyEntry method save all fields from incoming entry to pool's entry and latest are used in following log events.

Any thoughts about it? Sorry for my bad english.

how to reconnect to logstash when logstash server resart or no-write long time

Add Travis CI

Need to add unit-tests + functional/integration tests.

Is there a way to connect to Logstash over HTTP?

tcp: too many colons in address https://listener.logz.io:8071

Is there any other way to specify https?

I have been using http and I get this error:

Failed to fire hook: write tcp 192.168.1.36:55963->52.21.71.179:8070: write: broken pipe

So would it be possible to configure an additional delimiter for the hook? For example a nullcharacter?
This could be easily done like this

func (f LogstashFormatter) Format(e *logrus.Entry) ([]byte, error) {
    ne := copyEntry(e, f.Fields)
    dataBytes, err := f.Formatter.Format(ne)
    releaseEntry(ne)
    dataBytes = append(dataBytes, []byte("\000")...)
    return dataBytes, err
}

What do you think about it?

Thanks
eloo