logstasher version

1.0.0

OS version

All

Description

• Motivation
  We should be sure that after some changes the structure of data is expected and can be sent by different transports.
• Proposal
  Create test UDP/TCP servers with fake logstash data storing and add those logic into common tests.

Current behavior

UDP/TCP servers are missing in common tests.

Expected behavior

UDP/TCP servers are created and used in common tests.

As discussed in the forum https://erlangforums.com/t/logstasher-is-erlang-logger-formatter-for-logstash/1026/2

Just applying term_to_binary to any list that happens to be a valid iolist is not always intuitive

The one trying to log smth like that:

Tags = ["foo", "bar", "baz"],
logger:info(#{label => something_happened, tags => Tags})

would rater expect it to be logged as

{"label": "something_happened", "tags": ["foo", "bar", "baz"]}

than

{"label": "something_happened", "tags": "foobarbaz"}

In our project we ended up with smth like

term_to_json([C | _] = List) when is_integer(C) ->
    case io_lib:char_list(S) of
      true ->
        unicode:characters_to_binary(List, utf8);  %% string() or specific form of unicode:chardata()
      false ->
        lists:map(fun term_to_json/1, List)  % JS array
    end;

Another issue is that calling iolist_to_binary is not guaranteed to succeed if it's guarded by io_lib:char_list:

> S = "привет", io_lib:char_list(S).       
true
> iolist_to_binary(S).
** exception error: bad argument
     in function  iolist_to_binary/1
        called as iolist_to_binary([1087,1088,1080,1074,1077,1090])
        *** argument 1: not an iodata term

unicode:characters_to_binary(S, utf8) might be a better choice.

Add a back-off if there are TCP connection or data-send problems.

Proposal:

• Retry 5 times
• Backoff for configurable amount of time (default ca. 1 minute?)

And optionally:

• Buffer last N messages

logstasher version

1.0.0

OS version

Any

Description

• Motivation
  When running system is under stress load and it logs a lot, there is a risk that singletone logstasher process won't be able to process all the incoming log requests. In this scenario gen_server:call

would timeout which would cause OTP logger to disable this handler.

Also, for some latency-critical systems it might be unacceptable to hung for up to 5s on synchronous log calls.

This might be less of a problem for UDP connection, but since logstasher supports tcp, it may block for quite some time.

• Proposal
  Handlers in standard OTP logger provide flexible overload protection mechanisms.
  Unfortunately, they are not yet available as a library, so, AFAIK, other logger handler libraries tend to implement their own overload protection:

https://github.com/seriyps/logger_journald/blob/34b5d9cc0c35b06abe05fe2b5b7d826bf69fd8bc/src/logger_journald_h.erl#L120

https://github.com/hauleth/erlang-systemd/blob/8637dadf493d1a520d56cdd74db9813c5b86d669/src/systemd_journal_h.erl#L383

So I think it might be a good idea to add some sort of overload protection to logstasher too.

Current behavior

With TCP connection and high amount of logs, calls to logger:log may take up to 5s and logger handler may crash with timeout and being automatically disabled.

Expected behavior

Some sort of configurable overload protection mechanism. Ideally, with configuration compatible with OTP standard logger overload protection settings https://www.erlang.org/doc/apps/kernel/logger_chapter.html#overload_protection

Config

none