fluent / fluent-bit Goto Github PK
View Code? Open in Web Editor NEWFast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
Home Page: https://fluentbit.io
License: Apache License 2.0
Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
Home Page: https://fluentbit.io
License: Apache License 2.0
Segfault is randomly happened.
However it seems to be happen after outputting this log and it's id >= 2.
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=3
$ bin/fluent-bit -i health://google.com:80 -o stdout -vvv
Fluent-Bit v0.10.0
Copyright (C) Treasure Data
[2016/11/18 22:54:28] [ info] [engine] started
[2016/11/18 22:54:28] [debug] [router] default match rule health.0:stdout.0
[2016/11/18 22:54:29] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:30] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:31] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:32] [debug] [input] destroy input_thread id=0
[0] health.0: [1479477269, {"alive"=>true}]
[1] health.0: [1479477270, {"alive"=>true}]
[2] health.0: [1479477271, {"alive"=>true}]
[3] health.0: [1479477272, {"alive"=>true}]
[2016/11/18 22:54:33] [debug] [task] created task=0x1dc9cf0 id=0 OK
[2016/11/18 22:54:35] [debug] [task] destroy task=0x1dc9cf0 (task_id=0)
[2016/11/18 22:54:35] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:35] [debug] [input] destroy input_thread id=1
[2016/11/18 22:54:36] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:37] [debug] [input] destroy input_thread id=0
[0] health.0: [1479477275, {"alive"=>true}]
[1] health.0: [1479477275, {"alive"=>true}]
[2] health.0: [1479477276, {"alive"=>true}]
[3] health.0: [1479477277, {"alive"=>true}]
[2016/11/18 22:54:38] [debug] [task] created task=0x1dc81e0 id=0 OK
[2016/11/18 22:54:38] [debug] [task] destroy task=0x1dc81e0 (task_id=0)
[2016/11/18 22:54:38] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:39] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:40] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:41] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:42] [debug] [input] destroy input_thread id=0
[0] health.0: [1479477278, {"alive"=>true}]
[1] health.0: [1479477279, {"alive"=>true}]
[2] health.0: [1479477280, {"alive"=>true}]
[3] health.0: [1479477281, {"alive"=>true}]
[4] health.0: [1479477282, {"alive"=>true}]
[2016/11/18 22:54:43] [debug] [task] created task=0x1dc1f20 id=0 OK
[2016/11/18 22:54:47] [debug] [task] destroy task=0x1dc1f20 (task_id=0)
[2016/11/18 22:54:49] [debug] [task] created task=0x1dc2320 id=0 OK
[0] health.0: [1479477289, {"alive"=>true}]
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=0
[2016/11/18 22:54:49] [debug] [task] destroy task=0x1dc2320 (task_id=0)
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=2
[2016/11/18 22:54:49] [debug] [input] destroy input_thread id=3
Segmentation fault
Add NATS output support
http://fluentbit.io/news/ is not found.
It is linked from http://fluentbit.io/
Eventually some plugins would need "pre_exit" hook. In case of in_serial plugin, it would be better to have a chance to restore original UART parameters.
the current NATS output plugin do not implement the heartbeat check, this is useful to maintain a keepalive TCP connection.
We can pass a argument 'nanoseconds' to flb_input_set_collector_time().
But the value seems not to be used.(Only 'seconds' is used.)
I wrote a patch and it seemed to work.
But I modified some lib/monkey files which come from Monkey project.
# Changed but not updated:
...
# modified: ../lib/monkey/mk_core/include/mk_core/mk_event.h
# modified: ../lib/monkey/mk_core/mk_event.c
# modified: ../lib/monkey/mk_core/mk_event_epoll.c
# modified: ../src/flb_engine.c
So, which do you prefer?
Thanks.
I tested filter plugin and I found it causes segfault in some cases.
These commands cause segfault. Because "match" is NULL.
$ fluent-bit -i cpu -o stdout -F stdout
and
$ fluent-bit -i cpu -o stdout -F stdout -m '*'
Is option '-m' for only output plugins?
Grep plugin also causes.
$ fluent-bit -i cpu -o stdout -F grep
Illegal property (regex=aaa in this case)causes segfault.
This is a no regular expression error.
$ fluent-bit -i cpu -o stdout -F grep -p "match=*" -p "regex=aaa"
The jsmn library used had a bug and Fluent Bit code have a workaround for it. As soon as jsmn becomes fixed we should upgrade:
Please. So it can be used as a lightweight agent to collect from docker containers for example.
Fluent Bit 0.7 is adding tag routing, we need to add wildcards support for the matches instead of support just static string match.
Now, flb_output_new
and flb_input_new
return a pointer of plugin instance.
Would you move them to flb_ctx_t
to hide from user ?
Like this.
struct flb_lib_ctx {
struct mk_event_loop *event_loop;
struct mk_event *event_channel;
struct flb_config *config;
struct flb_input_instance *in;
struct flb_output_instance *out;
};
User specify plugin instance with plugin id (like fd
).
int in_id = 0;
...
in_id = flb_input_new(ctx, "mem", NULL); /* In this function, generate a instance and set ctx->in[in_id] to the instance */
flb_input_set(in_id, ctx, "tag", "test", NULL); /* set ctx->in[in_id]->properties to "tag" */
...
flb_destroy(ctx); /* release all of ctx->in */
Pros:
flb_ctx_t
. User doesn't need to release plugin instances. Engine will do.flb_stop
or flb_destroy
. (Now, it depends on user code.)Cons:
How about this?
It would be great to have Fluent Bit in homebrew, but they prefer to have clearly defined versions.
Could you please tag a release?
Or, it need more work to stable and it cannot to tag a stable release for now?
Development version of Fluent Bit allows to set optional TLS encryption for network I/O operations. The out_fluentd plugin needs support for the secure_forward protocol which after the common TLS handshake and certs checking it performs an extra authentication described here:
https://github.com/tagomoris/fluent-plugin-secure-forward#handshake
Some optional properties that needs to be handled:
As Fluent Bit is becoming portable between Linux, OSX and BSD, it should expose some mechanism to whitelist the features allowed for the target system (some plugins may not work on all systems)
Fluent Bit needs to get filtering plugins support, so data form inputs can be modified.
First priority is to get a Kubernetes metadata filter.
On a high load test, if many retries are issued, the Scheduler creates a timer file descriptor for each one, but easily these retries could not be scheduled at some point if we run out of file descriptors.
[2016/11/02 10:41:03] [ Error] timerfd: Too many open files, errno=24 at /home/edsiper/coding/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:175
Possible workarounds:
The approach 1 will let the user same the issue before take some action, note that using timerfd() is good as we just get notifications upon a timeout, we don't query a list.
Approach 2 add some overhead as we need to iterate a list every second.
This needs more thinking...
It would be nice if some of the unsupported inputs were supported on these platforms;
I'm not a C developer, or a BSD developer, but I'm willing to start work on the in_tail
plugin--I need that one for a work project (collecting logs from a PfSense router). I'm not sure where to begin with this though, so it would be awesome if someone could get me pointed in the right direction.
P.S: I do know Golang fairly well, so I could probably be more useful there if that is a viable route.
golang
repository became master
and some commits were lost.
https://github.com/fluent/fluent-bit/network
My forked repo may help us...
nokute78@73d9cc7
At https://github.com/fluent/fluent-bit/blob/master/plugins/out_http/http.c#L394 you only check for http return code 200 but all 2xx codes mean success per https://httpstatuses.com, in particular my application returns 204.
Would you like to add more error handling for return values from functions like the following?
In flb_error.h, some defines are octal (not decimal)
Is it intentional?
If not, I think there is a risk of overlap in the future.
Note:
As you know, the number which starts '0' is octal in C language.
For example, this value is 8 (not 10).
#define FLB_ERR_CFG_FILE 010
The function "flb_debug" does not belong to the list of async-signal-safe functions.
How do you think about to delete its call from your function "flb_signal_handler"?
in_cpu
outputs records one time only.
A variable flush_done
causes this issue.
$ bin/fluent-bit -i cpu -o stdout -vvv
Fluent-Bit v0.9.0
Copyright (C) Treasure Data
[2016/08/22 21:15:11] [ info] [engine] started
[2016/08/22 21:15:11] [debug] [router] default match rule cpu.0:stdout.0
[2016/08/22 21:15:12] [trace] [in_cpu] CPU 8.00%
[2016/08/22 21:15:13] [trace] [in_cpu] CPU 11.00%
[2016/08/22 21:15:14] [trace] [in_cpu] CPU 8.00%
[2016/08/22 21:15:15] [trace] [in_cpu] CPU 2.00%
[2016/08/22 21:15:16] [trace] [engine dispatch] task created 0x1d166b0
[2016/08/22 21:15:16] [trace] [thread 0x1d16780] created
[0] cpu.0: [1471868112, {"cpu_p"=>8.000000, "user_p"=>2.000000, "system_p"=>6.000000, "cpu0.p_cpu"=>8.000000, "cpu0.p_user"=>2.000000, "cpu0.p_system"=>6.000000}]
[1] cpu.0: [1471868113, {"cpu_p"=>11.000000, "user_p"=>4.000000, "system_p"=>7.000000, "cpu0.p_cpu"=>11.000000, "cpu0.p_user"=>4.000000, "cpu0.p_system"=>7.000000}]
[2] cpu.0: [1471868114, {"cpu_p"=>8.000000, "user_p"=>1.000000, "system_p"=>7.000000, "cpu0.p_cpu"=>8.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>7.000000}]
[3] cpu.0: [1471868115, {"cpu_p"=>2.000000, "user_p"=>1.000000, "system_p"=>1.000000, "cpu0.p_cpu"=>2.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>1.000000}]
[2016/08/22 21:15:16] [trace] [engine] [task event] task_id=0 thread_id=0 return=OK
[2016/08/22 21:15:16] [trace] [thread] destroy thread_id=0
[2016/08/22 21:15:16] [trace] [engine] destroy task_id=0
(no output)
Problem: fluent-bit
process maxes out a CPU core (100% usage). My build environment is the Raspberry Pi 1 (ARM). Followed the compile instructions in the README.
This is the command:
head events.json | ~/src/fluent-bit/build/bin/fluent-bit -i stdin -o stdout -f 1 > outtest
This only sends 10 events through fluent-bit. It should be idle after flushing those. The actual result is the high CPU usage.
Run strace
on the pid and I see the following lines being repeated while the process maxes out the CPU:
read(5, "", 32766) = 0
epoll_wait(3, {{EPOLLHUP, {u32=3363536, u64=4298330832}}}, 256, -1) = 1
read(5, "", 32766) = 0
epoll_wait(3, {{EPOLLHUP, {u32=3363536, u64=4298330832}}}, 256, -1) = 1
read(5, "", 32766) = 0
epoll_wait(3, {{EPOLLHUP, {u32=3363536, u64=4298330832}}}, 256, -1) = 1
...
I found this issue when I tested #48.
My environment is FreeBSD 10.2 , gcc 4.8.5. clang 3.4.1.
There are 2 issues.
I want to know how we fix issue1
This is compatibility issue.
This is caused these codes at lib/monkey/mk_core/mk_thread.c.
(Monkey project has same issue.)
#if defined (__linux__)
#include <ucontext.h>
#elif defined (__APPLE__)
#include <sys/ucontext.h>
#endif
There is no 'else' case.
It is very simple to replace #elif to #else or add FreeBSD define.
On the other hand, I found these articles.
That header is removed on 'POSIX issue7' .
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap13.html
OpenBSD doesn't have ucontext.h
https://groups.google.com/forum/#!topic/v8-dev/seLPS6grC2A
They recommended to replace these API to POSIX threads.
http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap03.html
getcontext(), makecontext(), swapcontext()
Due to portability issues with these functions, especially with the manipulation of contexts, applications are recommended to be rewritten to use POSIX threads.
[ 3%] Built target msgpack-static
[ 4%] Built target jsmn
[ 4%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_thread.c.o
fluent-bit/lib/monkey/mk_core/mk_thread.c:60:5: error: unknown type name 'ucontext_t'
ucontext_t context;
^
fluent-bit/lib/monkey/mk_core/mk_thread.c:72:5: error: unknown type name 'ucontext_t'
ucontext_t main;
^
fluent-bit/lib/monkey/mk_core/mk_thread.c:258:5: warning: implicit declaration of function 'swapcontext' is invalid in C99 [-Wimplicit-function-declaration]
swapcontext(&dt->context, &sch->main);
^
fluent-bit/lib/monkey/mk_core/mk_thread.c:287:9: warning: implicit declaration of function 'getcontext' is invalid in C99 [-Wimplicit-function-declaration]
getcontext(&dt->context);
^
fluent-bit/lib/monkey/mk_core/mk_thread.c:287:9: warning: declaration of built-in function 'getcontext' requires inclusion of the header <ucontext.h> [-Wbuiltin-requires-header]
fluent-bit/lib/monkey/mk_core/mk_thread.c:299:9: warning: implicit declaration of function 'makecontext' is invalid in C99 [-Wimplicit-function-declaration]
makecontext(&dt->context, (void (*)(void))_mk_thread_entry_point, 1, sch);
^
4 warnings and 2 errors generated.
*** Error code 1
Stop.
make[2]: stopped in fluent-bit/build
*** Error code 1
Stop.
make[1]: stopped in fluent-bit/build
*** Error code 1
Stop.
make: stopped in fluent-bit/build
out_flowcounter doesn't consider multiple inputs.
In this case, out_flowcounter counts data only from in_cpu.
I'm thinking how fix it...
$ bin/fluent-bit -i cpu -i mem -i random -o flowcounter -m '*'
Fluent-Bit v0.10.0
Copyright (C) Treasure Data
[2016/12/22 20:48:35] [ info] [engine] started
[out_flowcounter] cpu.0:[1482407375, {"counts":178, "bytes":10982, "counts/minute":2, "bytes/minute":183 }]
[out_flowcounter] cpu.0:[1482407435, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
[out_flowcounter] cpu.0:[1482407495, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
[out_flowcounter] cpu.0:[1482407555, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
[out_flowcounter] cpu.0:[1482407615, {"counts":180, "bytes":11040, "counts/minute":3, "bytes/minute":184 }]
Sbuffer of out_forward
seems to be leaked when fluent-bit can't communicate with server.
This issue is caused by cb_forward_flush
.
In some cases (=FLB_OUTPUT_RETURN(FLB_RETRY)), sbuffer is not destroyed.
Will it be released when fluent-bit will be able to communicate with server?
$ cmake .. -DFLB_MTRACE=yes -DFLB_DEBUG=yes && make
$ MALLOC_TRACE=hoge.txt bin/fluent-bit -i cpu -o forward -f 1
$ mtrace bin/fluent-bit hoge.txt
mtrace reports so many sbuffer are leaked.
$ mtrace bin/fluent-bit hoge.txt
Memory not freed:
-----------------
Address Size Caller
0x00002b93c00008c0 0x15 at 0x3a32217cd2
0x00002b93c00008e0 0x496 at 0x3a3220aebf
0x00002b93c0000d80 0x15 at 0x3a3220b174
0x00002b93c0000da0 0x38 at 0x3a3220cd5d
0x00002b93c0000de0 0x138 at 0x3a3220ff34
0x0000000000db2150 0x240 at 0x3a32211a83
0x0000000000db23a0 0xf at 0x3a32681022
0x0000000000db23c0 0x14 at 0x3a3269cf82
0x0000000000db23e0 0x14 at 0x3a3269cf82
0x0000000000db2400 0x15 at 0x3a3269cf82
0x0000000000db2420 0x14 at 0x3a3269cf82
0x0000000000db2600 0xd6 at 0x3a3269e91c
0x0000000000db6500 0 at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/include/fluent-bit/flb_mem.h:53
0x0000000000db6810 0x2000 at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/lib/msgpack-c-068041f/include/msgpack/sbuffer.h:77
0x0000000000db8820 0x2000 at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/lib/msgpack-c-068041f/include/msgpack/sbuffer.h:77
.
.
0x0000000000fa9b50 0x2000 at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/lib/msgpack-c-068041f/include/msgpack/sbuffer.h:77
$ bin/fluent-bit -i mem -o stdout
[2015/06/02 18:41:02] [ info] starting engine
[0] {"time"=>1433292063, "total"=>8081592, "free"=>1595860}
[1] {"time"=>1433292064, "total"=>8081592, "free"=>1596028}
[2] {"time"=>1433292065, "total"=>8081592, "free"=>1595900}
[3] {"time"=>1433292066, "total"=>8081592, "free"=>1600272}
[4] 0
[5] 0
[6] 0
[7] 0
[8] 0
[9] 0
[10] 0
[11] 0
[12] 33
[13] 1
[14] 0
[15] 0
[16] 0
[17] 0
[18] 0
[19] 0
[20] 104
[21] -19
[22] 86
[23] "d^?^@^@hVd^?^@^@^@"
[24] 0
[25] 0
[26] 0
[27] 0
[28] 0
Maybe I'm making a n00b mistake, but it looks like neither 0.10.0
or master
will compile on FreeBSD, regardless of version;
root@FreeBSD-10:~/fluent-bit/build # cmake ..
-- The C compiler identification is Clang 3.4.1
-- The CXX compiler identification is Clang 3.4.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/CC
-- Check for working CXX compiler: /usr/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.8")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test MSGPACK_ENABLE_GCC_CXX_ATOMIC
-- Performing Test MSGPACK_ENABLE_GCC_CXX_ATOMIC - Failed
-- Performing Test FLB_HAVE_SYNC_SUB
-- Performing Test FLB_HAVE_SYNC_SUB - Success
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Performing Test HAVE_STAT_H
-- Performing Test HAVE_STAT_H - Success
-- Performing Test HAVE_SYS_UIO_H
-- Performing Test HAVE_SYS_UIO_H - Success
-- Performing Test HAVE_UNISTD_H
-- Performing Test HAVE_UNISTD_H - Success
-- Performing Test HAVE_EPOLL
-- Performing Test HAVE_EPOLL - Failed
-- Performing Test HAVE_KQUEUE
-- Performing Test HAVE_KQUEUE - Failed
-- Event loop backend > select(2)
-- Performing Test HAVE_TIMERFD_CREATE
-- Performing Test HAVE_TIMERFD_CREATE - Failed
-- Performing Test HAVE_EVENTFD
-- Performing Test HAVE_EVENTFD - Failed
-- Found Perl: /usr/local/bin/perl (found version "5.20.3")
Can't open perl script "/root/fluent-bit/scripts/config.pl": No such file or directory
-- Performing Test FLB_HAVE_UCONTEXT
-- Performing Test FLB_HAVE_UCONTEXT - Success
-- Performing Test FLB_HAVE_VALGRIND
-- Performing Test FLB_HAVE_VALGRIND - Failed
-- Performing Test FLB_HAVE_FORK
-- Performing Test FLB_HAVE_FORK - Success
-- Performing Test FLB_HAVE_C_TLS
-- Performing Test FLB_HAVE_C_TLS - Success
-- Performing Test FLB_HAVE_SETJMP
-- Performing Test FLB_HAVE_SETJMP - Success
-- Performing Test FLB_HAVE_ACCEPT4
-- Performing Test FLB_HAVE_ACCEPT4 - Success
-- Performing Test FLB_HAVE_INOTIFY
-- Performing Test FLB_HAVE_INOTIFY - Failed
-- Looking for accept4
-- Looking for accept4 - found
-- Configuring done
CMake Warning (dev) at src/CMakeLists.txt:153 (add_library):
Policy CMP0063 is not set: Honor visibility properties for all target
types. Run "cmake --help-policy CMP0063" for policy details. Use the
cmake_policy command to set the policy and suppress this warning.
Target "fluent-bit-static" of type "STATIC_LIBRARY" has the following
visibility properties set for C:
C_VISIBILITY_PRESET
For compatibility CMake is not honoring them for this target.
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) at src/proxy/go/CMakeLists.txt:4 (add_library):
Policy CMP0063 is not set: Honor visibility properties for all target
types. Run "cmake --help-policy CMP0063" for policy details. Use the
cmake_policy command to set the policy and suppress this warning.
Target "flb-plugin-proxy-go" of type "STATIC_LIBRARY" has the following
visibility properties set for C:
C_VISIBILITY_PRESET
For compatibility CMake is not honoring them for this target.
This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done
-- Build files have been written to: /root/fluent-bit/build
root@FreeBSD-10:~/fluent-bit/build # make
Scanning dependencies of target msgpackc-static
[ 0%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/objectc.c.o
[ 1%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/unpack.c.o
[ 1%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/version.c.o
[ 2%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/vrefbuffer.c.o
[ 2%] Building C object lib/msgpack-c-0b7cabd/CMakeFiles/msgpackc-static.dir/src/zone.c.o
[ 3%] Linking C static library ../../library/libmsgpackc.a
[ 3%] Built target msgpackc-static
Scanning dependencies of target jsmn
[ 4%] Building C object lib/jsmn/CMakeFiles/jsmn.dir/jsmn.c.o
[ 4%] Linking C static library ../../library/libjsmn.a
[ 4%] Built target jsmn
Scanning dependencies of target ejson
[ 5%] Building C object lib/ejson/CMakeFiles/ejson.dir/ejson.c.o
/root/fluent-bit/lib/ejson/ejson.c:167:20: warning: passing 'unsigned char *' to parameter of type 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
ret = snprintf(buf + ctx->buf_pos,
^~~~~~~~~~~~~~~~~~
/usr/include/stdio.h:282:32: note: passing argument to parameter here
int snprintf(char * __restrict, size_t, const char * __restrict,
^
1 warning generated.
[ 5%] Linking C static library ../../library/libejson.a
[ 5%] Built target ejson
Scanning dependencies of target mk_core
[ 6%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_rconf.c.o
[ 6%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_string.c.o
[ 7%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_memory.c.o
[ 7%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_event.c.o
[ 8%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_utils.c.o
[ 8%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_file.c.o
[ 8%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_iov.c.o
[ 9%] Linking C static library ../../../library/libmk_core.a
[ 9%] Built target mk_core
Scanning dependencies of target mbedcrypto
[ 9%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/aes.c.o
[ 10%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/aesni.c.o
[ 10%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/arc4.c.o
[ 11%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/asn1parse.c.o
[ 11%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/asn1write.c.o
[ 12%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/base64.c.o
[ 12%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/bignum.c.o
[ 13%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/blowfish.c.o
[ 13%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/camellia.c.o
[ 14%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ccm.c.o
[ 14%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/cipher.c.o
[ 15%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/cipher_wrap.c.o
[ 15%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/cmac.c.o
[ 15%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ctr_drbg.c.o
[ 16%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/des.c.o
[ 16%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/dhm.c.o
[ 17%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecdh.c.o
[ 17%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecdsa.c.o
[ 18%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecjpake.c.o
[ 18%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecp.c.o
[ 19%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ecp_curves.c.o
[ 19%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/entropy.c.o
[ 20%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/entropy_poll.c.o
[ 20%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/error.c.o
[ 21%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/gcm.c.o
[ 21%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/havege.c.o
[ 21%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/hmac_drbg.c.o
[ 22%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md.c.o
[ 22%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md2.c.o
[ 23%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md4.c.o
[ 23%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md5.c.o
[ 24%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/md_wrap.c.o
[ 24%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/memory_buffer_alloc.c.o
[ 25%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/oid.c.o
[ 25%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/padlock.c.o
[ 26%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pem.c.o
[ 26%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pk.c.o
[ 27%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pk_wrap.c.o
[ 27%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkcs12.c.o
[ 28%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkcs5.c.o
[ 28%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkparse.c.o
[ 28%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/pkwrite.c.o
[ 29%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/platform.c.o
[ 29%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/ripemd160.c.o
[ 30%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/rsa.c.o
[ 30%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/sha1.c.o
[ 31%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/sha256.c.o
[ 31%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/sha512.c.o
[ 32%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/threading.c.o
[ 32%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/timing.c.o
[ 33%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/version.c.o
[ 33%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/version_features.c.o
[ 34%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedcrypto.dir/xtea.c.o
[ 34%] Linking C static library ../../../library/libmbedcrypto.a
[ 34%] Built target mbedcrypto
Scanning dependencies of target mbedx509
[ 34%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/certs.c.o
[ 35%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/pkcs11.c.o
[ 35%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509.c.o
[ 35%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_create.c.o
[ 36%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_crl.c.o
[ 36%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_crt.c.o
[ 37%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509_csr.c.o
[ 37%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509write_crt.c.o
[ 38%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedx509.dir/x509write_csr.c.o
[ 38%] Linking C static library ../../../library/libmbedx509.a
[ 38%] Built target mbedx509
Scanning dependencies of target mbedtls
[ 38%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/debug.c.o
[ 39%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/net_sockets.c.o
[ 39%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_cache.c.o
[ 40%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_ciphersuites.c.o
[ 40%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_cli.c.o
[ 41%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_cookie.c.o
[ 41%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_srv.c.o
[ 42%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_ticket.c.o
[ 42%] Building C object lib/mbedtls-2.4.1/library/CMakeFiles/mbedtls.dir/ssl_tls.c.o
[ 43%] Linking C static library ../../../library/libmbedtls.a
[ 43%] Built target mbedtls
Scanning dependencies of target sqlite3
[ 43%] Building C object lib/sqlite-amalgamation-3150200/CMakeFiles/sqlite3.dir/sqlite3.c.o
[ 44%] Linking C static library ../../library/libsqlite3.a
[ 44%] Built target sqlite3
Scanning dependencies of target cJSON
[ 44%] Building C object lib/cjson/CMakeFiles/cJSON.dir/src/cJSON.c.o
[ 44%] Linking C static library ../../library/libcJSON.a
[ 44%] Built target cJSON
Scanning dependencies of target co
[ 45%] Building C object lib/flb_libco/CMakeFiles/co.dir/libco.c.o
[ 45%] Linking C static library ../../library/libco.a
[ 45%] Built target co
Scanning dependencies of target flb-plugin-proxy-go
[ 46%] Building C object src/proxy/go/CMakeFiles/flb-plugin-proxy-go.dir/go.c.o
[ 46%] Linking C static library ../../../library/libflb-plugin-proxy-go.a
[ 46%] Built target flb-plugin-proxy-go
Scanning dependencies of target fluent-bit-static
[ 46%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_lib.c.o
[ 47%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_log.c.o
[ 47%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_uri.c.o
[ 47%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_pack.c.o
[ 48%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_sha1.c.o
[ 48%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_pipe.c.o
[ 49%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_kernel.c.o
[ 49%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_input.c.o
[ 50%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_output.c.o
[ 50%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_config.c.o
[ 51%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_network.c.o
[ 51%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_utils.c.o
[ 52%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_engine.c.o
[ 52%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_engine_dispatch.c.o
[ 53%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_task.c.o
[ 53%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_scheduler.c.o
[ 54%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_io.c.o
[ 54%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_upstream.c.o
[ 54%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_router.c.o
[ 55%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_http_client.c.o
[ 55%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_worker.c.o
[ 56%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_io_tls.c.o
[ 56%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_plugin_proxy.c.o
[ 57%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_sqldb.c.o
[ 57%] Linking C static library ../library/libfluent-bit.a
[ 57%] Built target fluent-bit-static
Scanning dependencies of target flb-plugin-out_flowcounter
[ 57%] Building C object plugins/out_flowcounter/CMakeFiles/flb-plugin-out_flowcounter.dir/out_flowcounter.c.o
[ 58%] Linking C static library ../../library/libflb-plugin-out_flowcounter.a
[ 58%] Built target flb-plugin-out_flowcounter
Scanning dependencies of target flb-plugin-out_lib
[ 58%] Building C object plugins/out_lib/CMakeFiles/flb-plugin-out_lib.dir/out_lib.c.o
[ 59%] Linking C static library ../../library/libflb-plugin-out_lib.a
[ 59%] Built target flb-plugin-out_lib
Scanning dependencies of target flb-plugin-out_td
[ 60%] Building C object plugins/out_td/CMakeFiles/flb-plugin-out_td.dir/td_http.c.o
In file included from /root/fluent-bit/plugins/out_td/td_http.c:28:
/root/fluent-bit/plugins/out_td/miniz/miniz.c:1012:14: warning: unused function 'def_realloc_func' [-Wunused-function]
static void *def_realloc_func(void *opaque, void *address, size_t items, size_t size) { (void)opaque, (void)address, (void)items, (void)size; return MZ_REALLOC(address, items * size); }
^
1 warning generated.
[ 60%] Building C object plugins/out_td/CMakeFiles/flb-plugin-out_td.dir/td_config.c.o
[ 61%] Building C object plugins/out_td/CMakeFiles/flb-plugin-out_td.dir/td.c.o
[ 61%] Linking C static library ../../library/libflb-plugin-out_td.a
[ 61%] Built target flb-plugin-out_td
Scanning dependencies of target flb-plugin-out_stdout
[ 62%] Building C object plugins/out_stdout/CMakeFiles/flb-plugin-out_stdout.dir/stdout.c.o
[ 62%] Linking C static library ../../library/libflb-plugin-out_stdout.a
[ 62%] Built target flb-plugin-out_stdout
Scanning dependencies of target flb-plugin-out_plot
[ 62%] Building C object plugins/out_plot/CMakeFiles/flb-plugin-out_plot.dir/plot.c.o
/root/fluent-bit/plugins/out_plot/plot.c:163:13: warning: implicit declaration of function 'dprintf' is invalid in C99 [-Wimplicit-function-declaration]
dprintf(fd, "%lu %" PRIu64 "\n", atime, val->via.u64);
^
1 warning generated.
[ 62%] Linking C static library ../../library/libflb-plugin-out_plot.a
[ 62%] Built target flb-plugin-out_plot
Scanning dependencies of target flb-plugin-out_null
[ 62%] Building C object plugins/out_null/CMakeFiles/flb-plugin-out_null.dir/null.c.o
[ 63%] Linking C static library ../../library/libflb-plugin-out_null.a
[ 63%] Built target flb-plugin-out_null
Scanning dependencies of target flb-plugin-out_influxdb
[ 64%] Building C object plugins/out_influxdb/CMakeFiles/flb-plugin-out_influxdb.dir/influxdb_bulk.c.o
[ 64%] Building C object plugins/out_influxdb/CMakeFiles/flb-plugin-out_influxdb.dir/influxdb.c.o
[ 65%] Linking C static library ../../library/libflb-plugin-out_influxdb.a
[ 65%] Built target flb-plugin-out_influxdb
Scanning dependencies of target flb-plugin-out_http
[ 66%] Building C object plugins/out_http/CMakeFiles/flb-plugin-out_http.dir/http.c.o
[ 66%] Linking C static library ../../library/libflb-plugin-out_http.a
[ 66%] Built target flb-plugin-out_http
Scanning dependencies of target flb-plugin-out_forward
[ 66%] Building C object plugins/out_forward/CMakeFiles/flb-plugin-out_forward.dir/__/__/src/flb_network.c.o
[ 67%] Building C object plugins/out_forward/CMakeFiles/flb-plugin-out_forward.dir/forward.c.o
[ 67%] Linking C static library ../../library/libflb-plugin-out_forward.a
[ 67%] Built target flb-plugin-out_forward
Scanning dependencies of target flb-plugin-out_es
[ 67%] Building C object plugins/out_es/CMakeFiles/flb-plugin-out_es.dir/es_bulk.c.o
[ 68%] Building C object plugins/out_es/CMakeFiles/flb-plugin-out_es.dir/es.c.o
[ 68%] Linking C static library ../../library/libflb-plugin-out_es.a
[ 68%] Built target flb-plugin-out_es
Scanning dependencies of target flb-plugin-out_counter
[ 68%] Building C object plugins/out_counter/CMakeFiles/flb-plugin-out_counter.dir/counter.c.o
[ 69%] Linking C static library ../../library/libflb-plugin-out_counter.a
[ 69%] Built target flb-plugin-out_counter
Scanning dependencies of target flb-plugin-in_random
[ 70%] Building C object plugins/in_random/CMakeFiles/flb-plugin-in_random.dir/random.c.o
[ 70%] Linking C static library ../../library/libflb-plugin-in_random.a
[ 70%] Built target flb-plugin-in_random
Scanning dependencies of target flb-plugin-in_forward
[ 71%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw.c.o
/root/fluent-bit/plugins/in_forward/fw.c:91:41: warning: incompatible pointer types passing 'int (struct flb_config *, void *)' to parameter of type 'int (*)(struct flb_input_instance *, struct flb_config *, void *)' [-Wincompatible-pointer-types]
in_fw_collect,
^~~~~~~~~~~~~
/root/fluent-bit/include/fluent-bit/flb_input.h:451:42: note: passing argument to parameter 'cb_new_connection' here
int (*cb_new_connection) (struct flb_input_instance *,
^
/root/fluent-bit/plugins/in_forward/fw.c:124:21: warning: incompatible pointer types initializing 'int (*)(struct flb_input_instance *, struct flb_config *, void *)' with an expression of type 'int (struct flb_config *, void *)' [-Wincompatible-pointer-types]
.cb_collect = in_fw_collect,
^~~~~~~~~~~~~
2 warnings generated.
[ 71%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw_conn.c.o
[ 72%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw_prot.c.o
[ 72%] Building C object plugins/in_forward/CMakeFiles/flb-plugin-in_forward.dir/fw_config.c.o
[ 73%] Linking C static library ../../library/libflb-plugin-in_forward.a
[ 73%] Built target flb-plugin-in_forward
Scanning dependencies of target flb-plugin-in_lib
[ 74%] Building C object plugins/in_lib/CMakeFiles/flb-plugin-in_lib.dir/in_lib.c.o
[ 74%] Building C object plugins/in_lib/CMakeFiles/flb-plugin-in_lib.dir/__/__/src/flb_pack.c.o
[ 75%] Linking C static library ../../library/libflb-plugin-in_lib.a
[ 75%] Built target flb-plugin-in_lib
Scanning dependencies of target flb-plugin-in_mqtt
[ 75%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt.c.o
[ 76%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt_conn.c.o
[ 76%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt_prot.c.o
[ 77%] Building C object plugins/in_mqtt/CMakeFiles/flb-plugin-in_mqtt.dir/mqtt_config.c.o
[ 77%] Linking C static library ../../library/libflb-plugin-in_mqtt.a
[ 77%] Built target flb-plugin-in_mqtt
Scanning dependencies of target flb-plugin-in_tcp
[ 77%] Building C object plugins/in_tcp/CMakeFiles/flb-plugin-in_tcp.dir/tcp.c.o
[ 78%] Building C object plugins/in_tcp/CMakeFiles/flb-plugin-in_tcp.dir/tcp_conn.c.o
[ 78%] Building C object plugins/in_tcp/CMakeFiles/flb-plugin-in_tcp.dir/tcp_config.c.o
[ 79%] Linking C static library ../../library/libflb-plugin-in_tcp.a
[ 79%] Built target flb-plugin-in_tcp
Scanning dependencies of target flb-plugin-in_stdin
[ 80%] Building C object plugins/in_stdin/CMakeFiles/flb-plugin-in_stdin.dir/in_stdin.c.o
[ 80%] Building C object plugins/in_stdin/CMakeFiles/flb-plugin-in_stdin.dir/__/__/src/flb_pack.c.o
[ 81%] Linking C static library ../../library/libflb-plugin-in_stdin.a
[ 81%] Built target flb-plugin-in_stdin
Scanning dependencies of target flb-plugin-in_serial
[ 82%] Building C object plugins/in_serial/CMakeFiles/flb-plugin-in_serial.dir/in_serial.c.o
[ 82%] Building C object plugins/in_serial/CMakeFiles/flb-plugin-in_serial.dir/in_serial_config.c.o
[ 82%] Linking C static library ../../library/libflb-plugin-in_serial.a
[ 82%] Built target flb-plugin-in_serial
Scanning dependencies of target flb-plugin-in_health
[ 82%] Building C object plugins/in_health/CMakeFiles/flb-plugin-in_health.dir/health.c.o
[ 82%] Linking C static library ../../library/libflb-plugin-in_health.a
[ 82%] Built target flb-plugin-in_health
Scanning dependencies of target flb-plugin-in_head
[ 82%] Building C object plugins/in_head/CMakeFiles/flb-plugin-in_head.dir/in_head.c.o
[ 83%] Linking C static library ../../library/libflb-plugin-in_head.a
[ 83%] Built target flb-plugin-in_head
Scanning dependencies of target fluent-bit-shared
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_lib.c.o
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_log.c.o
[ 84%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_uri.c.o
[ 85%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_pack.c.o
[ 85%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_sha1.c.o
[ 86%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_pipe.c.o
[ 86%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_kernel.c.o
[ 87%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_input.c.o
[ 87%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_output.c.o
[ 88%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_config.c.o
[ 88%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_network.c.o
[ 89%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_utils.c.o
[ 89%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_engine.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_engine_dispatch.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_task.c.o
[ 90%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_scheduler.c.o
[ 91%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_io.c.o
[ 91%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_upstream.c.o
[ 92%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_router.c.o
[ 92%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_http_client.c.o
[ 93%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_worker.c.o
[ 93%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_io_tls.c.o
[ 94%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_plugin_proxy.c.o
[ 94%] Building C object src/CMakeFiles/fluent-bit-shared.dir/flb_sqldb.c.o
[ 95%] Linking C shared library ../lib/libfluent-bit.so
/usr/bin/ld: cannot find -ldl
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1
Stop.
make[2]: stopped in /root/fluent-bit/build
*** Error code 1
Stop.
make[1]: stopped in /root/fluent-bit/build
*** Error code 1
Stop.
make: stopped in /root/fluent-bit/build
root@FreeBSD-10:~/fluent-bit/build #
Could be based on https://github.com/edenhill/librdkafka, payload should be JSON or MessagePack.
Currently we have https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud for google cloud logging but It would be great if there was a fluent-bit plugin that could do the same
I would like to point out that an identifier like "__flb_config_verbose
" does eventually not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?
support HTTP input, payload as JSON or MessagePack.
note: work in process.
SIGSEGV is occured with this PoC.
PoC(shell script)
while :; do mosquitto_pub -m '{"key":"aaa"}' -t some/topic & done
Backtrace
pi@raspberrypi /tmp/fluent-bit/build $ gdb bin/fluent-bit
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /tmp/fluent-bit/build/bin/fluent-bit...done.
(gdb) run -i mqtt -o stdout
Starting program: /tmp/fluent-bit/build/bin/fluent-bit -i mqtt -o stdout
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Fluent-Bit v0.8.0
Copyright (C) Treasure Data
[2016/04/05 12:58:47] [ info] starting engine
[2016/04/05 12:58:47] [ info] MQTT Listen='0.0.0.0' TCP_Port=1883
[2016/04/05 12:58:47] [ info] [mqtt] binding 0.0.0.0:1883
[2016/04/05 12:58:52] [ warn] No input data
[2016/04/05 12:58:57] [ warn] No input data
[2016/04/05 12:59:02] [ warn] No input data
[2016/04/05 12:59:07] [ warn] No input data
[2016/04/05 12:59:12] [ warn] No input data
[2016/04/05 12:59:17] [ warn] No input data
[2016/04/05 12:59:22] [ warn] No input data
Program received signal SIGSEGV, Segmentation fault.
0x000a8ef0 in mk_event_add (loop=0x706f742f, fd=239, type=3, mask=1, data=0x1f7de8)
at /tmp/fluent-bit/lib/monkey/mk_core/mk_event.c:88
88 ctx = loop->data;
(gdb) bt
#0 0x000a8ef0 in mk_event_add (loop=0x706f742f, fd=239, type=3, mask=1, data=0x1f7de8)
at /tmp/fluent-bit/lib/monkey/mk_core/mk_event.c:88
#1 0x00026b38 in mqtt_conn_add (fd=239, ctx=0xd4690) at /tmp/fluent-bit/plugins/in_mqtt/mqtt_conn.c:103
#2 0x00026848 in in_mqtt_collect (config=0xd1008, in_context=0xd4690) at /tmp/fluent-bit/plugins/in_mqtt/mqtt.c:102
#3 0x0001b478 in flb_engine_handle_event (config=0xd1008, mask=1, fd=9) at /tmp/fluent-bit/src/flb_engine.c:223
#4 flb_engine_start (config=0xd1008) at /tmp/fluent-bit/src/flb_engine.c:358
#5 0x00017110 in main (argc=5, argv=0x7efff7f4) at /tmp/fluent-bit/src/fluent-bit.c:314
(gdb)
Here, 'loop' is broken by received data.
loop=0x706f742f and '0x706f742f' is 'pot/' in ASCII.
'pot/' comes from the argument of mosquitto_pub, some'/top'ic
$ mosquitto_pub -m '{"key":"aaa"}' -t some/topic
flb_test_elasticsearch
is not finished successfully.
At flb_output_init()
, a variable p->flags
is checked like bitmap.
if (p->flags & FLB_IO_TLS) {
ins->tls.context = flb_tls_context_new(ins->tls_verify,
On the other hand, FLB_IO_TLS
is defined like enum at flb_io.h
.
/* Network operation modes */
#define FLB_IO_TCP 1 /* use plain TCP */
#define FLB_IO_TLS 2 /* use TLS/SSL layer */
#define FLB_IO_OPT_TLS 3 /* use TCP and optional TLS */
#define FLB_IO_ASYNC 4 /* use async mode (depends on event loop) */
Which is proper ?
Now, if (p->flags & FLB_IO_TLS)
is true when p->flags is set FLB_IO_OPT_TLS (out_es sets like this).
An extra null pointer check is not needed in functions like the following.
Would you like to apply the following semantic patch to find more update candidates?
@Remove_unnecessary_pointer_checks@
expression x;
@@
-if (x)
free(x);
Hi there, thanks for the cool utility.
I'm trying to find out how big the buffer size is when reading from stdin. Let's say I have a 10M file and I just want to cat
that file through fluent bit to my target.
So I do: cat events.log | td-agent-bit -c td-agent-bit.conf
where the conf file is:
[SERVICE]
Flush 1
Daemon off
Log_Level debug
[INPUT]
Name stdin
Tag events
[OUTPUT]
Name forward
Match events
Host example.com
Port 24224
I would expect the entire 10M file to be sent through the pipeline, however the actual result is that I get:
TD-Agent Bit v0.8.3
Copyright (C) Treasure Data
[2016/07/25 16:27:12] [ info] starting engine
[2016/07/25 16:27:12] [debug] [router] match rule stdin.0:forward.0
[2016/07/25 16:27:14] [ warn] STDIN data incomplete, waiting for more data...
And only a tiny fraction of the file contents are delivered.
Now, I can slow down the pipe using the following:
cat events.log | pv -l -L 10 -q | td-agent-bit -c td-agent-bit.conf
But it still doesn't work as I'd expect, I get many more events delivered, but still just about 1/10th. And that's slowing the rate to 10 bytes / second.
Is there a configuration toggle for the size of the buffer? I would like it to keep buffering data up to a certain limit, bounded by the amount of RAM on the box, I guess.
Thanks!
Fluent Bit when installed from a package (RPM/Deb) should include an init script and a basic configuration file. By default do nothing until the config changes and the service is restarted, it must support:
I tested 15047b7 and Segfault occurs.
(By the way, Default log level is debug
in that commit. Is it intentional?)
On the other hand, segfault is not happened when output is out_null
.
It is caused by src/flb_task.c
.
Program received signal SIGSEGV, Segmentation fault.
0x0000000000417e53 in flb_task_retry_clean (task=0x6d3f90, data=0x3a3298f040)
at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/src/flb_task.c:124
124 o_ins = out_th->o_ins;
Full log is
$ gdb bin/fluent-bit
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit...done.
(gdb) run -i cpu -o stdout -f 1
Starting program: /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit -i cpu -o stdout -f 1
[Thread debugging using libthread_db enabled]
[New Thread 0x2aaaaacc1700 (LWP 27777)]
Fluent-Bit v0.9.0
Copyright (C) Treasure Data
[2016/10/03 22:07:00] [ info] [engine] started
[2016/10/03 22:07:00] [debug] [router] default match rule cpu.0:stdout.0
[2016/10/03 22:07:02] [debug] [task] created task=0x6d3f90 OK
[0] cpu.0: [1475500021, {"cpu_p"=>4.000000, "user_p"=>1.000000, "system_p"=>3.000000, "cpu0.p_cpu"=>4.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>3.000000}]
[2016/10/03 22:07:03] [debug] [task] created task=0x6d41f0 OK
[0] cpu.0: [1475500022, {"cpu_p"=>0.000000, "user_p"=>0.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>0.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>0.000000}]
[2016/10/03 22:07:04] [debug] [task] created task=0x6d4320 OK
[0] cpu.0: [1475500023, {"cpu_p"=>2.000000, "user_p"=>1.000000, "system_p"=>1.000000, "cpu0.p_cpu"=>2.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>1.000000}]
[2016/10/03 22:07:05] [debug] [task] created task=0x6d4470 OK
[0] cpu.0: [1475500024, {"cpu_p"=>1.000000, "user_p"=>1.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>0.000000}]
[2016/10/03 22:07:06] [debug] [task] created task=0x6d45e0 OK
[0] cpu.0: [1475500025, {"cpu_p"=>4.000000, "user_p"=>2.000000, "system_p"=>2.000000, "cpu0.p_cpu"=>4.000000, "cpu0.p_user"=>2.000000, "cpu0.p_system"=>2.000000}]
Program received signal SIGSEGV, Segmentation fault.
0x0000000000417e53 in flb_task_retry_clean (task=0x6d3f90, data=0x3a3298f040)
at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/src/flb_task.c:124
124 o_ins = out_th->o_ins;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb)
$ bin/fluent-bit -i cpu -o forward -f 1
This issue also happens with in_http
out_http
.
The point is "offline" not to communicate fluentd/http server.
out_th is NULL when segfault is happened.
src/flb_task.c:81
81 o_ins = out_th->o_ins;
$ gdb bin/fluent-bit
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit...done.
(gdb) run -i cpu -o http -vvv -f 1
Starting program: /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/build/bin/fluent-bit -i cpu -o http -vvv -f 1
[Thread debugging using libthread_db enabled]
[New Thread 0x2aaaaacc1700 (LWP 28315)]
Fluent-Bit v0.9.0
Copyright (C) Treasure Data
[2016/10/31 22:14:01] [ info] [engine] started
[2016/10/31 22:14:01] [error] [TLS] Invalid CA file: /etc/ssl/certs/ca-certificates.crt
[2016/10/31 22:14:01] [debug] [router] default match rule cpu.0:http.0
[2016/10/31 22:14:02] [trace] [in_cpu] CPU 1.00%
[2016/10/31 22:14:03] [debug] [task] created task=0x6ea280 OK
[2016/10/31 22:14:03] [trace] [engine dispatch] task #0 created 0x6ea280
[2016/10/31 22:14:03] [trace] [thread 0x6ea320] created (custom data at 0x6ea348, size=64
[2016/10/31 22:14:03] [trace] [upstream] connection in process
[2016/10/31 22:14:03] [trace] [in_cpu] CPU 5.00%
[2016/10/31 22:14:03] [trace] [engine] resuming thread=0x6ea320
[2016/10/31 22:14:03] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:03] [error] [out_http] no upstream connections available
[2016/10/31 22:14:03] [trace] [engine] [task event] task_id=0 thread_id=0 return=RETRY
[2016/10/31 22:14:03] [trace] [thread] destroy thread=0x6ea320
[2016/10/31 22:14:03] [debug] [sched] retry=0x6f2620 0 in 6 seconds
[2016/10/31 22:14:04] [debug] [task] created task=0x6f0610 OK
[2016/10/31 22:14:04] [trace] [engine dispatch] task #1 created 0x6f0610
[2016/10/31 22:14:04] [trace] [thread 0x6f05a0] created (custom data at 0x6f05c8, size=64
[2016/10/31 22:14:04] [trace] [upstream] connection in process
[2016/10/31 22:14:04] [trace] [in_cpu] CPU 6.00%
[2016/10/31 22:14:04] [trace] [engine] resuming thread=0x6f05a0
[2016/10/31 22:14:04] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:04] [error] [out_http] no upstream connections available
[2016/10/31 22:14:04] [trace] [engine] [task event] task_id=1 thread_id=0 return=RETRY
[2016/10/31 22:14:04] [trace] [thread] destroy thread=0x6f05a0
[2016/10/31 22:14:04] [debug] [sched] retry=0x6f0740 1 in 2 seconds
[2016/10/31 22:14:05] [debug] [task] created task=0x6f0800 OK
[2016/10/31 22:14:05] [trace] [engine dispatch] task #2 created 0x6f0800
[2016/10/31 22:14:05] [trace] [thread 0x6f06d0] created (custom data at 0x6f06f8, size=64
[2016/10/31 22:14:05] [trace] [upstream] connection in process
[2016/10/31 22:14:05] [trace] [in_cpu] CPU 3.00%
[2016/10/31 22:14:05] [trace] [engine] resuming thread=0x6f06d0
[2016/10/31 22:14:05] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:05] [error] [out_http] no upstream connections available
[2016/10/31 22:14:05] [trace] [engine] [task event] task_id=2 thread_id=0 return=RETRY
[2016/10/31 22:14:05] [trace] [thread] destroy thread=0x6f06d0
[2016/10/31 22:14:05] [debug] [sched] retry=0x6f0950 2 in 5 seconds
[2016/10/31 22:14:06] [trace] [thread 0x6f08e0] created (custom data at 0x6f0908, size=64
[2016/10/31 22:14:06] [trace] [upstream] connection in process
[2016/10/31 22:14:06] [debug] [task] created task=0x6f0b50 OK
[2016/10/31 22:14:06] [trace] [engine dispatch] task #3 created 0x6f0b50
[2016/10/31 22:14:06] [trace] [thread 0x6f05a0] created (custom data at 0x6f05c8, size=64
[2016/10/31 22:14:06] [trace] [upstream] connection in process
[2016/10/31 22:14:06] [trace] [in_cpu] CPU 3.00%
[2016/10/31 22:14:06] [trace] [engine] resuming thread=0x6f08e0
[2016/10/31 22:14:06] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:06] [error] [out_http] no upstream connections available
[2016/10/31 22:14:06] [trace] [engine] resuming thread=0x6f05a0
[2016/10/31 22:14:06] [error] [io] TCP connection failed: 127.0.0.1:80
[2016/10/31 22:14:06] [error] [out_http] no upstream connections available
[2016/10/31 22:14:06] [trace] [engine] [task event] task_id=1 thread_id=1 return=RETRY
[2016/10/31 22:14:06] [trace] [thread] destroy thread=0x6f08e0
[2016/10/31 22:14:06] [debug] [sched] retry=0x6f0740 1 in 20 seconds
[2016/10/31 22:14:06] [trace] [engine] [task event] task_id=3 thread_id=0 return=RETRY
[2016/10/31 22:14:06] [trace] [thread] destroy thread=0x6f05a0
[2016/10/31 22:14:06] [debug] [sched] retry=0x6f0a60 3 in 9 seconds
Program received signal SIGSEGV, Segmentation fault.
0x000000000041a615 in flb_task_retry_create (task=0x6ea280, data=0x0) at /home/taka/git/oss/pull_req/fluentbit_env/fluent-bit/src/flb_task.c:81
81 o_ins = out_th->o_ins;
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.192.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb)
Now, each input plugin packs their data.
I want to change it.
Engine will be able to add a data to msgpack before flush.
I want to add unique ID(like MAC address, hostname, UUID, etc) to msgpack.
If thousands of fluent-bit send data to one fluentd, user wants to know which fluent-bit sent the data.
ID doesn't depend on input plugin, but we add codes to all input plugins.
However, using new method, we only add them to engine.
It is useful to another data which doesn't depend on input plugin. (e.g timestamp)
the libxbee3 component have a memory leak, I filed an issue here:
in_serial plugin read a stream of bytes, so the only thing is that user need to specify a delimiter string so the plugin can know when a message ends and a new one begins.
But, adding delimiter to JSON format is not suitable.
The request is that in_serial supports JSON Format.
When a connection is issued to a NATS server, the client receives an INFO message as soon the connection is established, e.g:
{"server_id":"e1d84d38a2c8f11f4209f079d8c82203",
"version":"0.7.5",
"go": "go1.5.3",
"host":"0.0.0.0",
"port":4449,
"auth_required":true,
"ssl_required":true,
"tls_required":true,
"tls_verify":true,
"max_payload":1048576
}
so based on this information the client (out_nats) must be able to switch to TLS mode. Fluent Bit already supports TLS but we need a tweak to do:
We are using 0.5.9, we should upgrade to the latest one:
https://github.com/msgpack/msgpack-c/releases/download/cpp-1.1.0/msgpack-1.1.0.tar.gz
Hi
I read fluent-bit/debian/copyright file.
License: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");
...
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
On Debian systems, the complete text of the GNU General
Public License version 2 can be found in "/usr/share/common-licenses/Apache-2.0".
Hmm..., This sentence is funny.
GNU GPL is not Apache License 2.0
And, it was included in Apace License Section.
You should correct this file.
I suggest you inspect other copyright and license files.
I could not find any documentation on how to read a log file like how fluentD does to output to elasticsearch on fluent-bit. In my case, there's already a stdout that I'm processing using fluentD (in Kubernetes). I'm trying to read some additional log files inside a pod using a pod volume.
Is this feature supported? Is it been targeted for a release in the near future?
$ make
...
Scanning dependencies of target fluent-bit-bin
[ 81%] Building C object src/CMakeFiles/fluent-bit-bin.dir/fluent-bit.c.o
Linking C executable ../bin/fluent-bit
../library/libmk_core.a(mk_event.c.o): In function `mk_utils_libc_error':
mk_event.c:(.text+0x81): undefined reference to `pthread_getspecific'
collect2: error: ld returned 1 exit status
make[2]: *** [bin/fluent-bit] Error 1
make[1]: *** [src/CMakeFiles/fluent-bit-bin.dir/all] Error 2
make: *** [all] Error 2
flb_test_elasticsearch.cpp includes "data/json_es.h".
However that header file is not found.
I've created a docker container with fluent-bit + alpine linux, however I attempt to run the example from site to pipe the script to fluent-bit stdin but it isn't work. Fluent bit not receive the script output.
test.sh
#!/bin/sh
while :; do
echo -n "{\"key\": \"some value\"}"
sleep 1
done
test.sh | fluent-bit -i stdin -o stdout
On Ubuntu works fine, only on alpine that I have this issue.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.