suikast42 / logunifier Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 298 KB

Unify logformats for Loki

Dockerfile 0.51% Shell 0.43% Go 99.06%

logunifier's People

Contributors

Watchers

logunifier's Issues

CLF ( Common Log Format) parser

Compress processError_rawData with zlib or gzip

Replace the pattern match cache.

Acutallay a normal hashmap is used. This is a potential mem leak cancditate. Replace it with a cache impl that have TTL for chace entries
https://github.com/Yiling-J/theine-go

Evaluate alternative proto compiler

https://docs.buf.build/how-to/migrate-from-prototool

evalute nats micro

https://pkg.go.dev/github.com/nats-io/nats.go/micro

Nats Jetstream receiver

Use Jpath for ecs to proto parsing

Handle loki so far behind error

10:56AM ERR Can't push message error="rpc error: code = Code(400) desc = entry with timestamp 2023-01-02 01:12:01.367 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2023-01-02T08:42:16Z' for st
ream: {ingress=\"vector-docker\", level=\"debug\", namespace=\"default\", stack=\"observability\", task=\"connect-proxy-grafana\", task_group=\"grafana\", used_grok=\"CONNECT_LOG\"},\ntotal ignored: 1 out of 1"
10:56AM ERR Can't push message error="rpc error: code = Code(400) desc = entry with timestamp 2023-01-02 01:12:01.367 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2023-01-02T08:42:16Z' for st
ream: {ingress=\"vector-docker\", level=\"debug\", namespace=\"default\", stack=\"observability\", task=\"connect-proxy-grafana\", task_group=\"grafana\", used_grok=\"CONNECT_LOG\"},\ntotal ignored: 1 out of 1"
10:56AM ERR Can't push message error="rpc error: code = Code(400) desc = entry with timestamp 2023-01-02 01:12:01.368 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2023-01-02T08:42:16Z' for st
ream: {ingress=\"vector-docker\", level=\"debug\", namespace=\"default\", stack=\"observability\", task=\"connect-proxy-grafana\", task_group=\"grafana\", used_grok=\"CONNECT_LOG\"},\ntotal ignored: 1 out of 1"