Consider the following config file:

input {
  stdin {}
}
filter {
  mutate {
    add_field => {"one" => 1, "two" => 2}
  }
}
output {
  stdout {}
}

Starting logfan outputs following error message:

2016/07/13 09:57:17 ParseHash parse error unexpected token ',' expected one of LSTokenComment|LSTokenIdentifier|LSTokenRCurlyBrace|LSTokenString on line 7 col 28

Not really an issue more like an information:
I've created a Docker image for Logfan because I needed it for my own use. Maybe it's useful for somebody else too:

Github repo:
https://github.com/pteich/docker-logfan

Docker Hub:
https://hub.docker.com/r/pteich/logfan/

It is based on Alpine Linux so it's quite small and uses your pre-built binaries from Github.

Hello!
I tried to use winlogbeat with input-beats in bitfan but it doesn't work.
bitfan output:

[10.33.52.25:61701] Received unknown type (2J): <nil>
[10.33.52.25:61705] Received unknown type (2J): <nil>
[10.33.52.25:61710] Received unknown type (2J): <nil>
[10.33.52.25:61714] Received unknown type (2J): <nil>

winlogbeat config:

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System
output.elasticsearch:
  enabled: false
output.logstash:
  enabled: true
  hosts: ["testelastic:5044"]
  worker: 1
  compression_level: 0
  ssl.enabled: false
output.console:
  enabled: false
  pretty: true
logging.level: info
logging.to_files: true
logging.files:

bitfan config:

input {
  beats {
    port => 5044
  }
}

output {
  stdout {
    codec => rubydebug {}
  }
}

Hello!
Usually in Logstash I used this included variables "%{+YYYY.MM.dd}" for writing data by days. But now I suspect that it doesn't work in bitfan. It would be great to add them.

My config is:

input  { stdin{} }
filter {
    mutate {
      add_field => {"date"=>"%{+YYYY.MM.dd}"}
    }
}
output { stdout { codec => rubydebug {}}}

Logstash output is:

{
       "message" => "qwe",
      "@version" => "1",
    "@timestamp" => "2017-02-07T19:32:13.558Z",
          "host" => "testelastic",
          "date" => "2017.02.07"
}

bitfan output is:

&{
  "@timestamp": "2017-02-08T00:36:27.113+05:00",
  "date":       "%{+YYYY.MM.dd}",
  "host":       "idea-PC",
  "message":    "qwe",
  "tags":       []interface {}{},
}

Hello!

You have to create your way to save the state in sincedb-file in each plugin.
Maybe its better to have common unit in logfan for saving the state that all plugins can use?

First, this is a great project, thank you! I noticed a small config parser issue. I often test with -e directly on command line and don't add extra whitespace, so this came up the first time I tried running logfan and took me a bit of puzzling to figure out.

Only difference between below 2 configs is the extra space prior to right curly at end of input section:

 [root@devbox logfan]# ./logfan -e 'input { stdin {} } output { stdout { codec => rubydebug } }'
 2016/08/03 07:38:40 ready
 ^C
 2016/08/03 07:38:41 stopping...
 2016/08/03 07:38:45 Everything stopped gracefully. Goodbye!

 [root@devbox logfan]# ./logfan -e 'input { stdin {}} output { stdout { codec => rubydebug } }'
 2016/08/03 07:38:49 ERROR while using config  Setting 1 parse error unexpected token '{' expected 'LSTokenAssignment' on line 1 col 34

Add a new bitfan command "restart" to stop and start a pipeline.

mandatory param : pipeline name or ID

Should accepts a optional parameter as a location to the configuration file to use.

When this param exists, stop the pipeline and start a new pipeline with the given configuration location.

When this param does not exist, stop the pipeline and reuse its configuration to restart the pipeline

New processor to search jira issues with a JQL or a filter ID.

may be used as an input and a filter to enrich an existing event with values.

Found issues should generate

• 1 resulting event with all result.
• or 1 event by issue.

Date typed fields should be converted to time.Time

Add a new section with documented working examples.

When http request fails Then a event should be raised with a tag _httppollfailure

This way this failure could be handle through a specific path in the pipeline to alert someone

Refactor base processor and processors to use a common option struct

Save each started pipeline configurations as a new version.

New processor to define (create, update) an issue in jira.

Build an URL like /pipeline/uri instead of /pipeline/plugin/uri

Env variables can be used in configuration files with ${ENV_NAME}.

Ui should provide an screen to manage Env variables globally available to pipelines. I thinking about passwords and host names.

Proposition

• An api to define variable and lookup defined variables
• • variables typed as “secret” can not see their values retrieved with api.
• • store variables in core database.
• • load Bitfan variables on start. And modify them at runtime with API.

problem found with a json response

Bitfan should reload last used pipelines on restart.

With an option to force Bitfan to start with no pipelines.

Short-circuiting is implemented in gopkg.in/Knetic/govaluate.v3.
Please update it.

Could you please explain how to use memory & webhook attributes - and what are they supposed to do, please? Thank you!

SQLite require CgO and is hard to cross compile.

Thinking about an unstructured DB, like filesystem or embeded nosql

A feature to accelerate pipeline authoring.

I’m thinking about a kind of playground which allow user to write some part of a configuration and get a live feedback displaying the produced event.

This could be used to test / debug part of a conf and save them as snippet to reuse in real configuration pipelines.

This playground should allow user to set the event to use as input (as a json) and display the produced event while typing into editor.

Suggestion : processors may handle a special option like a dry-run to bypass some instructions related to ressources state.
I’m thinking about input like “file” which save cursor in sincedb or filters like filter-change.

Existing implementation force to push collected data to a subfield

Allow result to be saved in event with transposed( first column values become the array row header)

Bitfan has no tests I think it is very important. When I write tests I find a lot of bugs.
I think it will be good to add https://coveralls.io to project at first.

And I want to know how you looks on http://goconvey.co?

Use this pattern to match data

<img([^>]*)src=['"]data:image/(\w+);base64,([^"']*)['"]([^>]*)>

Hello!
My usual case is receiving tab-separated line, split, and manipulate items. But in bitfan I can't use items of array after mutate split.

For example:
config:

input {
    stdin {}
}
filter {
    mutate {
        split => {"message" => " "}
    }
    mutate {
        add_field => {
            "one" => "%{message[0]}",
            "two" => "%{message[1]}",
            "three" => "%{message.2}",
            "four" => "%{[message][3]}"
        }
    }
}
output {
  stdout {
     codec => rubydebug { }
  }
}

output:

$ bitfan run
1 2 3 4
{
  "@timestamp": 2017-12-14 13:46:34 Local,
  "tags":       []string{},
  "one":        "[1 2 3 4]",
  "two":        "",
  "three":      "",
  "four":       "",
  "host":       "vm-bitfan1",
  "message":    []string{
    "1",
    "2",
    "3",
    "4",
  },
}

Probably it is specific of "github.com/clbanning/mxj", this library supports only maps but not slices.

Use bitfan as a layer between journalbeat and elasticsearch. Bitfan added unwanted by me new fields to log-messages. I am sure that these fileds is not present in journalbeat output and guilty is bitfan. New filed "message":"" appear if my config looks like this:

input {
  beats {
    host => ""
    port => 5044
  }
}
output {
  elasticsearch2 {
    host => "localhost"
    port => 9200
  }
  file {
    path => "/tmp/bitfan"
  }
}

And new filed "tags":[] appear if add mutate filter to config:

filter{
  mutate {
    convert => {"PRIORITY" => "integer"} 
  }
}

What's wrong with it? How I can remove it?

Count =2 send only one event

If you use { in if condition the bitfan stuck on start.

example:
filter {
if "message" =~ "^{" {
#if "{" in [message] { -> this one doesn't help
...
}
}

workaround:
filter {
eval {
expressions => { "msg_is_json" => "( [message] =~ '^{' ) ? true : false" }
}
if [msg_is_json] {
...
}
}

Hello! Thanks for project!

I am trying to add support to logfan to work as windows service, but I don't understand how to doit with cobra library.
If cobra is used only for short flags, so can it be replaced by pflag library of the same author?

When too much pipelines exist, it’s hard to find which ones belongs to alerts, reports, JIRA, data quality checker..., webhooks

Some of them are in dev other in production.

Labels attached to pipelines may allow filtering / organisation.

The following configuration grokfails because of LOGLEVEL log-level alias, when alias is loglevel it works well

input

2016-07-11T23:56:42.000+00:00 INFO [MySecretApp.com.Transaction.Manager]:Starting transaction for session -464410bf-37bf-475a-afc0-498e0199f008

filter

grok {
   match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level}" }
 }

grok processor should accept dashbased alias

Allows sprint format %{fieldName} in to, cc, bcc, replyto

Hello @AlexAkulov !

Please when your are ready to push your http-output, use this repo instead of veino/...
I deleted veino organisation as it does not mean anything

Now :

• processors and runtime belong to the same repo
• runtime has been refactored to use only channels only instead of maps.
• processors hava no dependency with runtime "core"

Processor's behavior changed a bit.

Now only one processor starts and its Receive(e) method is called concurrently up to the number of workers.

When a processor cannot accept concurent call it should mention it with a MaxConcurent() int function which returns the max concurrent call the processor can accept.

• return 0 to fix no limit
• return 1 to force processor for receive one event at a time

See processors/filter-digest for an example.

Please, add ES6 support - gopkg.in/olivere/elastic.v6

ERRO[0000] [core] output_elasticsearch2 Agent '10': Can not configure agent output_elasticsearch2 : no Elasticsearch node available component=core

When used as filter, http don't reuse received event, so it lost event's values after processing

I don't understand why project is split into five different repositories.
I wanted to build package of logfan from my fork and this made a lot of problems for me.
Maybe it will be better to unit all repositories into one?

go version go1.6.3 darwin/amd64

go get
# github.com/vjeantet/logfan/cmd
../vjeantet/logfan/cmd/start.go:39: runtime.Logger().SetVerboseMode undefined (type *runtime.vienoLogger has no field or method SetVerboseMode)

Add new params

• to choose http headers to be sent back to client
• to choose what to send as body response (Send generated events ID as default)

As a user I want to export and backup pipelines configuration files

The graal ! :-)

A feature

• to share a pipeline/snippet with a community.
• to import a pipeline/snippet shared by someone.

Allow embedding images in email

Please add support multiline codec for syslog.