kr / logfmt Goto Github PK

values := map[string]string{}
logfmt.Unmarshal(log_content, &values)

The code will crash; must I use a specified struct?

Before I raise a PR, any thoughts as to why not?

A follow-up on https://gist.github.com/kr/0e8d5ee4b954ce604bb2 really. I would be interested in helping out and make logfmt more official and supported. Is there some interest in this ?

At this point there is an EBNFish syntax and a reference implementation. I think there are still clarifications to be made, for example the encoding of the content (I would say binary), if there is a limit on the length of the line (I would say it depends on the transport but 4k should be a safe bet). Ideally there would exist a test suite to help other languages implement the parsing and production of this format and an official website.

Let me know if I can help with any of these (except the website, I suck at design). Cheers !

The spec specifies that values containing bytes less than or equal to " " (\x20) must be quoted but not escaped.
Specifically with regards to newlines this is rather problematic when parsing incomplete logstreams. Take this example:

...
msg=" level=info time=2021-07-30T08:24:01.604Z
msg="
...

Apologies for the rather contrived scenario, but it shows the ambiguity when parsing a logstream rather well. The point here is you don't know whether a newline is the beginning, middle, or end of a log entry. The complete output for the above record(s) could look like this:

msg=" level=info time=2021-07-30T08:24:01.604Z
msg="

So: {msg: " level=info time=2021-07-30T08:24:01.604Z\nmsg="}
Or this:

out="
msg=" level=info time=2021-07-30T08:24:01.604Z
msg="
" level=error time=2021-07-30T08:24:05.004Z

So: {out: "\nmsg=", level: "info", time: "2021-07-30T08:24:01.604Z"} and {msg: "\n", level: "error", time: "2021-07-30T08:24:05.004Z"}

The bigger issue here is that there is no way to get back to a "clean slate". The second version could have variations that never cause a parsing error and the parser would keep on having an off-by-one error:

...
moreLines x=1 f="
out="
msg=" level=info time=2021-07-30T08:24:01.604Z
msg="
" level=error time=2021-07-30T08:24:05.004Z
x="is this quoted, who knows
" it might be. It all depends on the first line.

The above could be a long string value, or it could be a bunch of keys with the value true

It seems that the default delimiter used is (space).
I'm using the logfmt parser in fluent-bit, but have a use case where i need the delimiter to be , (comma space).
Is this a built-in functionality of the logfmt package?

Below is a sample log file which contains a message field inside the json array

{"timestamp":"2021-05-26T20:45:08.751Z","sequence":568,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.keycloak.events","level":"WARN","message":"type=LOGIN_ERROR, realmId=master, clientId=ui, userId=92cdda0b-28b4-46df-8f81-48458708eb48, ipAddress=1.1.1.1, error=invalid_user_credentials, auth_method=openid-connect, auth_type=code, redirect_uri=https://centralus-v2.mydomain.com/signin, code_id=f8d8be37-eb0e-40cc-a507-db5862cd0719, username=[email protected], authSessionParentId=f8d8be37-eb0e-40cc-a507-db5862cd0719, authSessionTabId=pvMeYvDFhU4","threadName":"default task-2","threadId":147,"mdc":{},"ndc":"","hostName":"keycloak-b9576f6db-np5dx","processName":"jboss-modules.jar","processId":617}

I adopted the code in the example into a main package and expected the program to produce no errors. However, when I ran the program, I received this:

$ go version
go version go1.0.3

$ go run x.go
2013/04/30 18:22:30 err="invalid syntax"

Here is x.go:

package main

import (
    "bytes"
    "fmt"
    "github.com/kr/logfmt"
    "log"
    "strconv"
)

type Measurement struct {
    Key  string
    Val  float64
    Unit string // (e.g. ms, MB, etc)
}

type Measurements []*Measurement

var measurePrefix = []byte("measure.")

func (mm *Measurements) HandleLogfmt(key, val []byte) error {
    if !bytes.HasPrefix(key, measurePrefix) {
        return nil
    }
    i := bytes.LastIndexFunc(val, isDigit)
    v, err := strconv.ParseFloat(string(val[:i+1]), 10)
    if err != nil {
        return err
    }
    m := &Measurement{
        Key:  string(key[len(measurePrefix):]),
        Val:  v,
        Unit: string(val[i+1:]),
    }
    *mm = append(*mm, m)
    return nil
}

// return true if r is an ASCII digit only, as opposed to unicode.IsDigit.
func isDigit(r rune) bool {
    return '0' <= r && r <= '9'
}

func main() {
    var data = []byte("measure.a=1ms measure.b=10 measure.c=100MB measure.d=1s garbage")

    mm := make(Measurements, 0)
    if err := logfmt.Unmarshal(data, &mm); err != nil {
        log.Fatalf("err=%q", err)
    }
    for _, m := range mm {
        fmt.Printf("%v\n", *m)
    }

}

It would help with distribution integration.
Thanks.

The README says:

Go package for parsing (and, eventually, generating) log lines in the logfmt style.

Is there still interest in working on the generating part?

I've been contributing to both log15 and gokit/log. log15 already supports a logfmt flavor of output, and gokit/log would like to. In go-kit/kit#34 I suggested it would be nice to have an encoding/json or encoding/xml style package that supports marshaling and unmarshaling logfmt data with support for the encoding.TextMarshaler and encoding.TextUnmarshaller interfaces.

Is there interest in doing that work here? If not, would @kr and @bmizerany object to a new project attempting to create such an API for logfmt, possibly starting as a fork of this repo? Would you be willing to help with feedback if questions arise regarding logfmt specifics?