Log Server

Log processing pipeline configurable with yaml files.

The pipeline is composed by nodes that pass the event data down the hierarchy. There are three classes of nodes:

• source: Generates event data. This can be a node that reads from a file, listens on a socket or directly generates events from a local source. It's also the node that parses the data into a dictionary of strings that will be consumed by the other nodes.

• action: This node has a parent (input event) and children. It can be used to transform the data, use it to do some calculations or filter the events used down the tree.

• sink: This node receives input events but does not forward it to any other node. It should be used for termination actions like saving events to a file or sending and email.

Events

Events are parsed into a map. For example, this line:

t=2017-11-14t23:10:03+0000 lvl=info msg="job started" module=borges worker=2 job=015fbccb-f1b2-2903-3aac-4eafc2196a7d caller=archiver.go:76

is parsed as:

t:      2017-11-14t23:10:03+0000
lvl:    info
msg:    job started
module: borges
worker: 2
job:    015fbccb-f1b2-2903-3aac-4eafc2196a7d caller=archiver.go:76

Nodes

• FileSource (source): Reads log lines from a file. It's only able to parse log15 format. The input file is specified in args.
• RegexpFilter (process): It filters events using regular expressions. Its input is a map specifying the key that has to check and as value is the regular expression it has to match. The filter map is specified in map.
• PrintKeySink (sink): Prints an specific event key. The key is specified in args.
• CounterAction (action): Creates a new counter named args and increments it each time an event passes the node.
• PrintCountersSink (sink): Prints all the counters each time it receives an event. It does not have arguments.

Example:

---
- name: reader
  type: FileSource
  args: logs
  children:
    - name: count all
      type: CounterAction
      args: total
      children:
      - name: filter siva error
        type: RegexpFilter
        map:
          lvl: "^eror$"
          error: "index read failed"
        children:
          - name: print root
            type: PrintKeySink
            args: "root"
          - name: print msg
            type: PrintKeySink
            args: "msg"
          - name: count siva errors
            type: CounterAction
            args: siva_errors
            children:
              - name: print counters
                type: PrintCountersSink