bshuster-repo / logrus-logstash-hook Goto Github PK
View Code? Open in Web Editor NEW:information_source: Logstash hook for logrus
License: MIT License
:information_source: Logstash hook for logrus
License: MIT License
I think is good to create a wiki to tell how to configure the logstash config files according to your filters and the current format.
Hi,
Hook and logrus can write logs with different set of fields.
For exmaple,
log.WithField("animal", "walrus"}).Info("First log")
log.Info("Second log")
Logrus output writes second log without animal field while hook will send log with animal equal to "walrus"
After inspecting hook's code I have found reason for this behavior. copyEntry method save all fields from incoming entry to pool's entry and latest are used in following log events.
Any thoughts about it? Sorry for my bad english.
Hi,
We need a new release to fix the bug with persistent fields added with WithField method
Thanks! :)
Sometimes (server restart, intermittent network outages) a TCP connection breaks, and is never restored unless the app is restarted or a hook is recreated anew.
I'd propose implementing automatic reconnections when possible (i.e. persist protocol
and address
in Hook
if they were provided). Easiest solution would be to just re-Dial
if h.conn.Write
fails, but since logging is synchronous and starting a TCP connection when the network is down may take a whole minute, it may essentially lock up the caller.
If desirable, I can work on a pull request that'd move network-sending code into a separate goroutine, which would receive data from a buffered channel (or, if introducing a new third-party dependency is acceptable, a RingChannel
).
What do you think?
Hi,
I've tried to convert my project to go modules.
By default the last release 0.41.0is chosen which is from 2017 and quite older than master.
I think it would be wise to create a new release from time to time.
Current workaround:
Update you go.mod file to use the master branch instead of version.
Need to add unit-tests + functional/integration tests.
Hello,
i'm testing your library right now to use it together with a fluentD stack and it looks like we could need an additional delimiter to the fired messages.
So would it be possible to configure an additional delimiter for the hook? For example a nullcharacter?
This could be easily done like this
func (f LogstashFormatter) Format(e *logrus.Entry) ([]byte, error) {
ne := copyEntry(e, f.Fields)
dataBytes, err := f.Formatter.Format(ne)
releaseEntry(ne)
dataBytes = append(dataBytes, []byte("\000")...)
return dataBytes, err
}
What do you think about it?
Thanks
eloo
how to reconnect to logstash when logstash server resart or no-write long time
This is an informal issue.
Since the v1.0.3 or loggrus, the commit 325575f18110626a8fd7f027f899aa8c8f5169c6 makes the v0.4 to fail because of the dependency of DefaultTimestampFormat
logstash_formatter.go
Note that the v1.0-beta is not impacted by this issue.
The uppercase "Sirupsen" in this project's import path collides with the lower-case variant.
When trying to compile on macOS this error occurs:
case-insensitive import collision: "github.com/Sirupsen/logrus" and "github.com/sirupsen/logrus"
I can not switch to an upper-case import path because other hooks I use, use the lower-case variant:
cannot use logger (type *"github.com/Sirupsen/logrus".Logger) as type *"github.com/sirupsen/logrus".Logger in assignment
Hello, I'm just opening this issue to let you know, the next release of logrus (v1.2.0) will add a new trace level named Trace below Debug.
Here is the PR for reference sirupsen/logrus#844
You may want to take into account this new level.
After long stretches of inactivity our logstash connection times out with the following errors:
Failed to fire hook: write tcp 10.0.0.212:45992->10.0.4.98:5000: write: connection timed out
Failed to fire hook: write tcp 10.0.0.212:45992->10.0.4.98:5000: write: broken pipe
As such, logs are no longer written. Is there any way to have the logstash hook automatically retry the connection?
Hello,
I'm testing your library just like your example but nothing written in LogStash/Elasticsearch until I add a one second time delay before the program ended!
here the code it is:
func main() {
log := logrus.New()
conn, err := net.Dial("tcp", "127.0.0.1:8911")
if err != nil || conn == nil {
log.Fatal(err)
}
hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "myappName"}))
log.Hooks.Add(hook)
ctx := log.WithFields(logrus.Fields{
"method": "main",
})
ctx.Info("Hello World in print again!")
time.Sleep(1 * time.Second)
}
I'll appreciate if anybody can help
How to change timestamp format from string
to Unix format float64
or int64
??
by default it gives "@timestamp" => "2016-02-29T16:57:23.000Z",
I want to pass time as Unix time "@timestamp" => 1611664414.310306
.
And if it possible to leave time
as is in time.RFC3339
format.
e.g. outgoing message should be like this
{
"@level": "info",
"@message": "Init logger with Logstash!",
"time": "2021-01-26T19:10:59+03:00",
"@timestamp": 1611677459.123
}
The only thing I succeed in is that I can add such timestamp in extra fields.
hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"hostname": "my_sginy_host", "@timestamp":time.Now().Unix()}))
logger.Hooks.Add(hook)
Such code adds @timestamp
in the fields
section.
{
"@timestamp": "2021-01-26T19:10:59+03:00",
"@version":"1",
"fields.@timestamp": 1611677459,
"hostname": "my_shiny_host",
"@level": "info",
"message": "Init.redis!", "type":"log"
}
Why the single commit b3d898b is tagged as 1.0, 0.5-beta, v5.0, v1.0 and v0.5 at the same time??
This is very confusing.
Hi there,
I've set up this hook, and the timestamps are at whole second intervals, so sorting on the timestamps where there are more than 1 per second is somewhat broken.
for the console logs that come out of logrus I was able to get accurate timestamps like this:
log := logrus.New()
fmt := new(logrus.TextFormatter)
fmt.TimestampFormat = time.RFC3339Nano
fmt.FullTimestamp = true
log.SetFormatter(fmt)
for this hook though I'm unable to figure out how to get more accurate timestamps
This is the console:
This is kibana, sorted by timestamp (out of order):
Is there a way around this?
I'm having problems to lock this package to a specific version because of the use of branches instead of releases, could you please create releases for them?
As I can see so far we would have 0.2
, 0.3
, 0.4
, 1.0
.
The @timestamp did not have microsecond. It's 2017-04-24 18:54:18.000.
I think microsecond is important like 2017-04-24 18:54:18.231
timeStampFormat = logrus.DefaultTimestampFormat
to
timeStampFormat = time.RFC3339Nano
package main
import (
"github.com/Sirupsen/logrus"
"github.com/bshuster-repo/logrus-logstash-hook"
)
func main() {
log := logrus.New()
hook, err := logrus_logstash.NewHook("tcp", "172.17.0.2:9999", "myappName")
if err != nil {
log.Fatal(err)
}
log.Hooks.Add(hook)
ctx := log.WithFields(logrus.Fields{
"method": "main",
})
ctx.Info("Hello World!")
}
errors with
# command-line-arguments
./logstash.go:16: cannot use hook (type *logrus_logstash.Hook) as type "..../vendor/github.com/Sirupsen/logrus".Hook in argument to log.Hooks.Add:
*logrus_logstash.Hook does not implement "..../vendor/github.com/Sirupsen/logrus".Hook (wrong type for Fire method)
have Fire(*"github.com/Sirupsen/logrus".Entry) error
want Fire(*"..../vendor/github.com/Sirupsen/logrus".Entry) error
What could be the problem?
This was a bit annoying but I found that the README that displays on github is out of date, the README found in vendor/github/bshuster-repo/logrus-logstash-hook is up to date however.
Its not wholly intuitive for someone to check the vendor files for help on installed packages.
I am trying making this logrus -> logstash
pipeline work. I was using filebeats as my logstash's input. But what should I use for logrus? A simple logstash pipeline config sample is as follows.
input {
# is there a "logrus" input for logrus?
beats {
port => 5044
}
}
#filter {
# should I do any filtering stuff here?
#}
output {
elasticsearch {
hosts => "192.168.0.1:9200"
manage_template => false
index => "logstash-gateway-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
thanks a lot!
It's just a topic for a discussion.
If I try to send a large message then it doesn't fit to MTU. This is from tcpdump
:
10:35:03.216961 IP XXX.XXX.XXX.XXX.55505 > XXX.XXX.XXX.XXX.3535: UDP, bad length 1914 > 1472
However it doesn't seems to be the problem of this project. It seems to be a problem of used net.Conn
โฆ Or UDP just cannot be used for large messages in logstash at all.
So the question: what is the simplest way (if exists) to use this plugin over UDP?
how can i set log level?
for example just error and fatal?
tcp: too many colons in address https://listener.logz.io:8071
Is there any other way to specify https?
I have been using http and I get this error:
Failed to fire hook: write tcp 192.168.1.36:55963->52.21.71.179:8070: write: broken pipe
Master Code and Release source code has different folder stucture
Works with standard go get, but not works with dep
Fedora Rawhide, with Go 1.14beta1, logrus-logstash-hook 0.4.1
Testing in: /builddir/build/BUILD/logrus-logstash-hook-0.4.1/_build/src
PATH: /builddir/build/BUILD/logrus-logstash-hook-0.4.1/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
GOPATH: /builddir/build/BUILD/logrus-logstash-hook-0.4.1/_build:/usr/share/gocode
GO111MODULE: off
command: go test -buildmode pie -compiler gc -ldflags "-X github.com/bshuster-repo/logrus-logstash-hook/version=0.4.1 -extldflags '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '"
testing: github.com/bshuster-repo/logrus-logstash-hook
github.com/bshuster-repo/logrus-logstash-hook
--- FAIL: TestLogstashFormatter (0.00s)
logstash_formatter_test.go:58: expected data[error] to be 'Get http://example.com: The error' but got 'Get "http://example.com": The error'
FAIL
Seem like the formatting changed slightly to include quotes.
After creating the connection with Dial, how the hook will be able to maintain the connection if it drops?
How it will reconnect?
In the next version (1.xx) I am going to do a major change around the API:
WithPrefix
, WithFields
and WithField
, NewHookWithConn
and etc will be removed.I try to set Report caller
Logger.SetReportCaller(true)
but caller report not send to logstash
Just something I noticed in your readme
The @Version field should really be a string:
https://github.com/logstash-plugins/logstash-codec-json/blob/master/spec/codecs/json_spec.rb#L26
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.