vjeantet / bitfan Goto Github PK
View Code? Open in Web Editor NEWData and Event processing pipeliner really inspired by Logstash
Home Page: https://bitfan.io
License: Apache License 2.0
Data and Event processing pipeliner really inspired by Logstash
Home Page: https://bitfan.io
License: Apache License 2.0
Allow result to be saved in event with transposed( first column values become the array row header)
Please, add ES6 support - gopkg.in/olivere/elastic.v6
ERRO[0000] [core] output_elasticsearch2 Agent '10': Can not configure agent output_elasticsearch2 : no Elasticsearch node available component=core
Add a new section with documented working examples.
Use this pattern to match data
<img([^>]*)src=['"]data:image/(\w+);base64,([^"']*)['"]([^>]*)>
Save each started pipeline configurations as a new version.
Short-circuiting is implemented in gopkg.in/Knetic/govaluate.v3.
Please update it.
problem found with a json response
Hello!
My usual case is receiving tab-separated line, split, and manipulate items. But in bitfan I can't use items of array after mutate split.
For example:
config:
input {
stdin {}
}
filter {
mutate {
split => {"message" => " "}
}
mutate {
add_field => {
"one" => "%{message[0]}",
"two" => "%{message[1]}",
"three" => "%{message.2}",
"four" => "%{[message][3]}"
}
}
}
output {
stdout {
codec => rubydebug { }
}
}
output:
$ bitfan run
1 2 3 4
{
"@timestamp": 2017-12-14 13:46:34 Local,
"tags": []string{},
"one": "[1 2 3 4]",
"two": "",
"three": "",
"four": "",
"host": "vm-bitfan1",
"message": []string{
"1",
"2",
"3",
"4",
},
}
Probably it is specific of "github.com/clbanning/mxj", this library supports only maps but not slices.
Hello!
You have to create your way to save the state in sincedb-file in each plugin.
Maybe its better to have common unit in logfan for saving the state that all plugins can use?
go version go1.6.3 darwin/amd64
go get
# github.com/vjeantet/logfan/cmd
../vjeantet/logfan/cmd/start.go:39: runtime.Logger().SetVerboseMode undefined (type *runtime.vienoLogger has no field or method SetVerboseMode)
When too much pipelines exist, it’s hard to find which ones belongs to alerts, reports, JIRA, data quality checker..., webhooks
Some of them are in dev other in production.
Labels attached to pipelines may allow filtering / organisation.
When used as filter, http don't reuse received event, so it lost event's values after processing
Bitfan has no tests I think it is very important. When I write tests I find a lot of bugs.
I think it will be good to add https://coveralls.io to project at first.
And I want to know how you looks on http://goconvey.co?
Hello!
I tried to use winlogbeat with input-beats in bitfan but it doesn't work.
bitfan output:
[10.33.52.25:61701] Received unknown type (2J): <nil>
[10.33.52.25:61705] Received unknown type (2J): <nil>
[10.33.52.25:61710] Received unknown type (2J): <nil>
[10.33.52.25:61714] Received unknown type (2J): <nil>
winlogbeat config:
winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: Security
- name: System
output.elasticsearch:
enabled: false
output.logstash:
enabled: true
hosts: ["testelastic:5044"]
worker: 1
compression_level: 0
ssl.enabled: false
output.console:
enabled: false
pretty: true
logging.level: info
logging.to_files: true
logging.files:
bitfan config:
input {
beats {
port => 5044
}
}
output {
stdout {
codec => rubydebug {}
}
}
I don't understand why project is split into five different repositories.
I wanted to build package of logfan from my fork and this made a lot of problems for me.
Maybe it will be better to unit all repositories into one?
Refactor base processor and processors to use a common option struct
Use bitfan as a layer between journalbeat and elasticsearch. Bitfan added unwanted by me new fields to log-messages. I am sure that these fileds is not present in journalbeat output and guilty is bitfan. New filed "message":"" appear if my config looks like this:
input {
beats {
host => ""
port => 5044
}
}
output {
elasticsearch2 {
host => "localhost"
port => 9200
}
file {
path => "/tmp/bitfan"
}
}
And new filed "tags":[] appear if add mutate filter to config:
filter{
mutate {
convert => {"PRIORITY" => "integer"}
}
}
What's wrong with it? How I can remove it?
New processor to define (create, update) an issue in jira.
New processor to search jira issues with a JQL or a filter ID.
may be used as an input and a filter to enrich an existing event with values.
Found issues should generate
Date typed fields should be converted to time.Time
Hello! Thanks for project!
I am trying to add support to logfan to work as windows service, but I don't understand how to doit with cobra library.
If cobra is used only for short flags, so can it be replaced by pflag library of the same author?
$ go get -u github.com/vjeantet/logstack
# github.com/veino/processors/input-twitter
veino/processors/input-twitter/twitter.go:98: cannot use api.PublicStreamFilter(v) (type *anaconda.Stream) as type anaconda.Stream in assignment
veino/processors/input-twitter/twitter.go:106: p.stream.Interrupt undefined (type anaconda.Stream has no field or method Interrupt)
veino/processors/input-twitter/twitter.go:107: p.stream.End undefined (type anaconda.Stream has no field or method End)
# github.com/vjeantet/logstack/parser
vjeantet/logstack/parser/lexer.go:6: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:27: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:66: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:94: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:110: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:133: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:172: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:184: undefined: lexer.IsDigit
vjeantet/logstack/parser/lexer.go:195: l.SkipWhitespace undefined (type *lexer.L has no field or method SkipWhitespace)
vjeantet/logstack/parser/lexer.go:200: undefined: lexer.IsLetter
vjeantet/logstack/parser/lexer.go:200: too many errors
Env variables can be used in configuration files with ${ENV_NAME}.
Ui should provide an screen to manage Env variables globally available to pipelines. I thinking about passwords and host names.
Not really an issue more like an information:
I've created a Docker image for Logfan because I needed it for my own use. Maybe it's useful for somebody else too:
Github repo:
https://github.com/pteich/docker-logfan
Docker Hub:
https://hub.docker.com/r/pteich/logfan/
It is based on Alpine Linux so it's quite small and uses your pre-built binaries from Github.
First, this is a great project, thank you! I noticed a small config parser issue. I often test with -e directly on command line and don't add extra whitespace, so this came up the first time I tried running logfan and took me a bit of puzzling to figure out.
Only difference between below 2 configs is the extra space prior to right curly at end of input section:
[root@devbox logfan]# ./logfan -e 'input { stdin {} } output { stdout { codec => rubydebug } }'
2016/08/03 07:38:40 ready
^C
2016/08/03 07:38:41 stopping...
2016/08/03 07:38:45 Everything stopped gracefully. Goodbye!
[root@devbox logfan]# ./logfan -e 'input { stdin {}} output { stdout { codec => rubydebug } }'
2016/08/03 07:38:49 ERROR while using config Setting 1 parse error unexpected token '{' expected 'LSTokenAssignment' on line 1 col 34
As a user I want to export and backup pipelines configuration files
Add new params
Count =2 send only one event
The following configuration grokfails because of LOGLEVEL log-level
alias, when alias is loglevel
it works well
input
2016-07-11T23:56:42.000+00:00 INFO [MySecretApp.com.Transaction.Manager]:Starting transaction for session -464410bf-37bf-475a-afc0-498e0199f008
filter
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level}" }
}
grok processor should accept dashbased alias
Build an URL like /pipeline/uri instead of /pipeline/plugin/uri
Please add support multiline codec for syslog.
A feature to accelerate pipeline authoring.
I’m thinking about a kind of playground which allow user to write some part of a configuration and get a live feedback displaying the produced event.
This could be used to test / debug part of a conf and save them as snippet to reuse in real configuration pipelines.
This playground should allow user to set the event to use as input (as a json) and display the produced event while typing into editor.
Suggestion : processors may handle a special option like a dry-run to bypass some instructions related to ressources state.
I’m thinking about input like “file” which save cursor in sincedb or filters like filter-change.
Consider the following config file:
input {
stdin {}
}
filter {
mutate {
add_field => {"one" => 1, "two" => 2}
}
}
output {
stdout {}
}
Starting logfan outputs following error message:
2016/07/13 09:57:17 ParseHash parse error unexpected token ',' expected one of LSTokenComment|LSTokenIdentifier|LSTokenRCurlyBrace|LSTokenString on line 7 col 28
Could you please explain how to use memory
& webhook
attributes - and what are they supposed to do, please? Thank you!
Allows sprint format %{fieldName} in to, cc, bcc, replyto
SQLite require CgO and is hard to cross compile.
Thinking about an unstructured DB, like filesystem or embeded nosql
When http request fails Then a event should be raised with a tag _httppollfailure
This way this failure could be handle through a specific path in the pipeline to alert someone
Hello!
Usually in Logstash I used this included variables "%{+YYYY.MM.dd}" for writing data by days. But now I suspect that it doesn't work in bitfan. It would be great to add them.
My config is:
input { stdin{} }
filter {
mutate {
add_field => {"date"=>"%{+YYYY.MM.dd}"}
}
}
output { stdout { codec => rubydebug {}}}
Logstash output is:
{
"message" => "qwe",
"@version" => "1",
"@timestamp" => "2017-02-07T19:32:13.558Z",
"host" => "testelastic",
"date" => "2017.02.07"
}
bitfan output is:
&{
"@timestamp": "2017-02-08T00:36:27.113+05:00",
"date": "%{+YYYY.MM.dd}",
"host": "idea-PC",
"message": "qwe",
"tags": []interface {}{},
}
If you use { in if condition the bitfan stuck on start.
example:
filter {
if "message" =~ "^{" {
#if "{" in [message] { -> this one doesn't help
...
}
}
workaround:
filter {
eval {
expressions => { "msg_is_json" => "( [message] =~ '^{' ) ? true : false" }
}
if [msg_is_json] {
...
}
}
Bitfan should reload last used pipelines on restart.
With an option to force Bitfan to start with no pipelines.
Add a new bitfan command "restart" to stop and start a pipeline.
mandatory param : pipeline name or ID
Should accepts a optional parameter as a location to the configuration file to use.
When this param exists, stop the pipeline and start a new pipeline with the given configuration location.
When this param does not exist, stop the pipeline and reuse its configuration to restart the pipeline
Hello @AlexAkulov !
Please when your are ready to push your http-output, use this repo instead of veino/...
I deleted veino organisation as it does not mean anything
Now :
Processor's behavior changed a bit.
Now only one processor starts and its Receive(e)
method is called concurrently up to the number of workers.
When a processor cannot accept concurent call it should mention it with a MaxConcurent() int
function which returns the max concurrent call the processor can accept.
See processors/filter-digest for an example.
Allow embedding images in email
The graal ! :-)
A feature
bitfan/processors/httppoller/httppoller.go
Line 193 in ed05016
Existing implementation force to push collected data to a subfield
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.