Giter VIP home page Giter VIP logo

syslogparser's Introduction

Syslogparser

This is a syslog parser for the Go programming language.

https://pkg.go.dev/github.com/jeromer/syslogparser

Installing

go get github.com/jeromer/syslogparser

Supported RFCs

Not all features described in RFCs above are supported but only the most part of it. For exaple SDIDs are not supported in RFC 5424 and STRUCTURED-DATA are parsed as a whole string.

This parser should solve 80% of use cases. If your use cases are in the 20% remaining ones I would recommend you to fully test what you want to achieve and provide a patch if you want.

Parsing an RFC 3164 syslog message

b := "<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8"
buff := []byte(b)

p := rfc3164.NewParser(buff)
err := p.Parse()
if err != nil {
	panic(err)
}

for k, v := range p.Dump() {
	fmt.Println(k, ":", v)
}

You should see

timestamp : 2013-10-11 22:14:15 +0000 UTC
hostname  : mymachine
tag       : su
content   : 'su root' failed for lonvick on /dev/pts/8
priority  : 34
facility  : 4
severity  : 2

Parsing an RFC 5424 syslog message

b := `<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry...`
buff := []byte(b)

p := rfc5424.NewParser(buff)
err := p.Parse()
if err != nil {
	panic(err)
}

for k, v := range p.Dump() {
	fmt.Println(k, ":", v)
}

You should see

version : 1
timestamp : 2003-10-11 22:14:15.003 +0000 UTC
app_name : evntslog
msg_id : ID47
message : An application event log entry...
priority : 165
facility : 20
severity : 5
hostname : mymachine.example.com
proc_id : -
structured_data : [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]

Detecting message format

You can use the DetectRFC() function. Like this:

b := []byte(`<165>1 2003-10-11T22:14:15.003Z ...`)
rfc, err := syslogparser.DetectRFC(b)
if err != nil {
	panic(err)
}

switch rfc {
case RFC_UNKNOWN:
	fmt.Println("unknown")
case RFC_3164:
	fmt.Println("3164")
case RFC_5424:
	fmt.Println("5424")
}

Running tests

Run make test

Running benchmarks

Run make benchmark

go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: github.com/jeromer/syslogparser
BenchmarkDetectRFC-8   	81994480	        14.7 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/jeromer/syslogparser	2.145s

cd rfc3164 && go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: github.com/jeromer/syslogparser/rfc3164
BenchmarkParseTimestamp-8   	 2823901	       416 ns/op	      16 B/op	       1 allocs/op
BenchmarkParseHostname-8    	34796552	        35.4 ns/op	      16 B/op	       1 allocs/op
BenchmarkParseTag-8         	20954252	        59.3 ns/op	       8 B/op	       1 allocs/op
BenchmarkParseHeader-8      	 2276569	       596 ns/op	      80 B/op	       3 allocs/op
BenchmarkParsemessage-8     	 6751579	       192 ns/op	     104 B/op	       4 allocs/op
BenchmarkParseFull-8        	 1445076	       838 ns/op	     336 B/op	      10 allocs/op
PASS

ok  	github.com/jeromer/syslogparser/rfc3164	9.601s
cd rfc5424 && go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: github.com/jeromer/syslogparser/rfc5424
BenchmarkParseTimestamp-8   	  790478	      1488 ns/op	     432 B/op	      21 allocs/op
BenchmarkParseHeader-8      	 1000000	      1043 ns/op	     336 B/op	      18 allocs/op
BenchmarkParseFull-8        	  980828	      1306 ns/op	     672 B/op	      21 allocs/op
PASS
ok  	github.com/jeromer/syslogparser/rfc5424	4.356s

syslogparser's People

Contributors

abligh avatar eenblam avatar jeromer avatar mcuadros avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

syslogparser's Issues

Parsing of time appears to be contrary to RFC3164

RFC3164 states (at 4.1.2):

The TIMESTAMP field is the local time and is in the format of "Mmm dd hh:mm:ss" (without the quote marks)

(my emphasis). rfc3164.go has the following code

ts, err = time.Parse(tsFmt, string(sub))

The documentation for time.Parse says:

In the absence of a time zone indicator, Parse returns a time in UTC.

If the local time of the syslog origin is not UTC, this will return an erroneous time. Using the local timezone of the syslog server interpreting the information is also incorrect, as that could be different from the local timezone of the syslog client. Obviously this is really a defect in the syslog protocol and sane people would set all their local timezones (for syslog purposes) to UTC, but apparently not everyone is sane.

It would therefore be useful to be able to specify the 'local' timezone of the server to the parser; if this is set, use time.ParseInLocation() else (as currently) use Parse().

I think this should be unnecessary for RFC5424 parsing.

can set 'parsePriority' to option?

on my linux host, the syslog:

Jul  3 10:48:10 Blackstone041201 sshd[2991]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul  3 10:48:10 Blackstone041201 sshd[2991]: subsystem request for sftp
Jul  3 10:48:10 Blackstone041201 sshd[2991]: subsystem request for sftp

so, can set 'parsePriority' to option:

func ParsePriority(buff []byte, cursor *int, l int) (Priority, error) {
	pri := newPriority(0)

	if l <= 0 {
		return pri, ErrPriorityEmpty
	}

	if buff[*cursor] != PRI_PART_START {
		//return pri, ErrPriorityNoStart   ------------ no priority segment
                return pri, nil
	}

	i := 1
	priDigit := 0

	for i < l {
		......
	}

	return pri, ErrPriorityNoEnd
}

Panic in rfc3164.go L192

Hi,

I'm using go-syslog (which imports this package) to receive logs from installing servers. I'm seeing a panic originating from rfc3164.go on line 192 when a blank message (?) arrives.

Here's a tcpdump:

11:32:09.659220 IP (tos 0x0, ttl 64, id 21646, offset 0, flags [DF], proto UDP (17), length 101)
    192.168.1.187.61000 > 192.168.1.5.5514: UDP, length 73
E..eT.@[email protected].
<13>Mar 18 11:32:10 log-output: HTTP request sent, awaiting response... 

11:32:09.666068 IP (tos 0x0, ttl 64, id 21647, offset 0, flags [DF], proto UDP (17), length 78)
    192.168.1.187.61000 > 192.168.1.5.5514: UDP, length 50
E..NT.@[email protected]...:.N<13>Mar 18 11:32:10 log-output: No data received.

11:32:09.666343 IP (tos 0x0, ttl 64, id 21648, offset 0, flags [DF], proto UDP (17), length 70)
    192.168.1.187.61000 > 192.168.1.5.5514: UDP, length 42
E..FT.@[email protected]..<13>Mar 18 11:32:10 log-output: Retrying.

11:32:09.666360 IP (tos 0x0, ttl 64, id 21649, offset 0, flags [DF], proto UDP (17), length 61)
    192.168.1.187.61000 > 192.168.1.5.5514: UDP, length 33
E..=T.@[email protected]...)..<13>Mar 18 11:32:10 log-output: 

In this example, the first three entries are parsed fine, but the last one causes the panic (paths truncated)

panic: runtime error: index out of range
goroutine 9 [running]:
panic(0xa68880, 0xc82000e020)
/.../build/go/src/runtime/panic.go:464 +0x3e6
.../github.com/jeromer/syslogparser/rfc3164.(*Parser).parseTag(0xc8200e4aa0, 0x0, 0x0, 0x0, 0x0)
/.../github.com/jeromer/syslogparser/rfc3164/rfc3164.go:192 +0x214
.../github.com/jeromer/syslogparser/rfc3164.(*Parser).parsemessage(0xc8200e4aa0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
.../github.com/jeromer/syslogparser/rfc3164/rfc3164.go:109 +0x78
.../github.com/jeromer/syslogparser/rfc3164.(*Parser).Parse(0xc8200e4aa0, 0x0, 0x0)
.../github.com/jeromer/syslogparser/rfc3164/rfc3164.go:56 +0x130
.../gopkg.in/mcuadros/go-syslog%2ev2.(*Server).parser(0xc820536240, 0xc8209c4000, 0x2a, 0x10000, 0xc82083dd00, 0x1
.../gopkg.in/mcuadros/go-syslog.v2/server.go:248 +0x92
.../gopkg.in/mcuadros/go-syslog%2ev2.(*Server).goParseDatagrams.func1(0xc820536240)
.../gopkg.in/mcuadros/go-syslog.v2/server.go:363 +0x282
created by .../gopkg.in/mcuadros/go-syslog%2ev2.(*Server).goParseDatagrams
.../gopkg.in/mcuadros/go-syslog.v2/server.go:367 +0x92

Thanks

Test suite broken

dependencies.txt specifies correct source for gocheck, but the test files themselves try to import from launchpad.

Error parsing RFC3164 messages without a timestamp

RFC3164 messages without a timestamp, such as those produced by some Ubiquiti devices, are detected as RFC3164 but then parsing fails with "Timestamp format unknown"

For example,

<14>MiniSwitch 7483c04f9d75,USW_FLEX_MINI-1.8.6.694: NETDEV: Setup PVID... done

where "MiniSwitch" is the hostname, returns the error "Timestamp format unknown"

Can we just return no timestamp instead of an error?

Parsing issue

Hello,
this kind of log is not parsed : <15> Sep 12 16:53:56 XXXTR555 This is a log.
Is it because of the space ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.