Server's identifier should be hostname instead of ip address about echofish HOT 9 CLOSED

Hi there, this is one of the areas we're looking at improving in the near future. Its not a simple decision to base the server identifier in either way.

The arguments for IP instead of hostname are mainly

• indexing (much faster to index integers than say varchar)
• storage (4bytes vs LengthOfHostname+1)

However, you do present a valid case, so maybe we could look into make this a configurable knob to switch between ip/host. Would that be an acceptable solution for you?

As a side note, would you be interested in sharing some details on the setups so that we could try to figure a workaround? How did you configure syslog on your docker images, what syslog daemon(s) you run on the sending side?

from echofish.

The arguments for IP instead of hostname are mainly
indexing (much faster to index integers than say varchar)
storage (4bytes vs LengthOfHostname+1)

You could add an auto_increment key as primary key in the host table. This column could be used as foreign key in others tables like archive and syslog.

However, you do present a valid case, so maybe we could look into make this a configurable knob to switch between ip/host. Would that be an acceptable solution for you?

This would be ok for me.

As a side note, would you be interested in sharing some details on the setups so that we could try to figure a workaround? How did you configure syslog on your docker images, what syslog daemon(s) you run on the sending side?

All servers are using rsyslog on Ubuntu servers. Containers are plain LXC (no docker), running also rsyslog. Servers and containers forwards the logs into one central dedicated rsyslog server, also running Ubuntu.

On all servers/containers, I have this in /etc/rsyslog.d/90-remote.conf:
*.* @@a.b.c.d:514

from echofish.

Hi,

I looked into ways that might get you around this situation, until we come up with a sane fix to combat this.

A syslog relay could be setup "in the middle" (on the host that runs the containers) and instructed to replace 'Host' with 'Source-IP'.

The loghost can then be configured to use the Host field (instead of 'Source-IP') for messages arriving from the relay.

Not a pretty solution, but i got it working in a test environment.

If you have many hosts running NAT-ed containers, you would need to setup a relay on each host and make sure all containers have unique ip address (even if they are contained in different hosts).

So in case it fits your needs, i will detail the relevant configuration (for rsyslog).

List of hosts:

• 192.168.0.2 A container running behind NAT.
• 10.0.0.2 A host running linux containers. In the private network 192.168.0.0/24, its IP is 192.168.0.1.
• 10.0.0.1 Echofish's rsyslog (the final destination inserting to mysql)

The client (192.168.0.2) forwards to the syslog relay (192.168.0.1):

*.* @192.168.0.1

The syslog relay (192.168.0.1 and 10.0.0.2) forwards logs to echofish (10.0.0.1):

$ModLoad imudp.so
$UDPServerRun 514

$template ReplaceHostWithIP, "<%pri%> %timestamp% %fromhost-ip% %syslogtag% %msg%\n"
$template ReplaceLocalhost, "<%pri%> %timestamp% 10.0.0.2 %syslogtag% %msg%\n"

# rsyslog loghost
if $fromhost-ip != '127.0.0.1' then @10.0.0.1:514;ReplaceHostWithIP
& ~
if $fromhost-ip == '127.0.0.1' then @10.0.0.1:514;ReplaceLocalhost

rsyslog config for the final echofish destination (10.0.0.1) must also be slightly modified to add a check for messages that arrive from the specific syslog relay (10.0.0.2):

# /etc/rsyslog.d/echofish.conf
# sql + rules for rsyslog integration with Echofish

# Load rsyslog MySQL plugin
$ModLoad ommysql.so

# Generic template
$template dbFormat,"INSERT INTO archive_bh (host, facility, priority, level, received_ts, program, msg,pid,tag) VALUES ( '%fromhost-ip%', '%syslogfacility%', '%syslogpriority%','%syslogseverity%', '%timereported:::date-mysql%', TRIM('%programname%'), TRIM('%msg%'),'', '%syslogtag%' );\n",sql

# Specific template for loghost (log self as 10.0.0.1, instead of 127.0.0.1)
$template dbFormatLocal,"INSERT INTO archive_bh (host, facility, priority, level, received_ts, program, msg,pid,tag) VALUES ( '10.0.0.1', '%syslogfacility%', '%syslogpriority%','%syslogseverity%', '%timereported:::date-mysql%', TRIM('%programname%'), TRIM('%msg%'),'', '%syslogtag%' );\n",sql

# Specific template for syslog relays (use %hostname%, it already has valid IPv4 addr)
$template dbFormatRelay,"INSERT INTO archive_bh (host, facility, priority, level, received_ts, program, msg,pid,tag) VALUES ( '%hostname%', '%syslogfacility%', '%syslogpriority%','%syslogseverity%', '%timereported:::date-mysql%', TRIM('%programname%'), TRIM('%msg%'),'', '%syslogtag%' );\n",sql

# IMPORTANT: change {{{echofish-pass-here}}} to match your setup !!!
if $fromhost-ip == '10.0.0.2' then :ommysql:127.0.0.1,ETS_echofish,echofish,{{{echofish-pass-here}}};dbFormatRelay
& ~
if $fromhost-ip != '127.0.0.1' then :ommysql:127.0.0.1,ETS_echofish,echofish,{{{echofish-pass-here}}};dbFormat
& ~
if $fromhost-ip == '127.0.0.1' then :ommysql:127.0.0.1,ETS_echofish,echofish,{{{echofish-pass-here}}};dbFormatLocal

Notice: All tests where conducted with rsyslogd started with flags -Q and -x. In debian, this involves changing the file /etc/defaults/rsyslogd to include the extra parameters when starting rsyslog.

from echofish.

Thanks for the workaround. I'll start by integrating my physicals servers first and look for the containers later. I already have so many rules to create anyway :)

from echofish.

Dear ohmer1,

We have pushed new code in master branch that should solve this among other issues. Your testing would be welcome. There is even an (un-tested) UPGRADE.md with upgrade instructions from 0.4.x to -current.

from echofish.

Hi,

Is it stable enough to upgrade my setup to master?

Thanks!

from echofish.

I was merely asking for your testing, since the latest code solves the hostname issue you reported.

It would be safer if you went for a fresh install before attempting to upgrade your production setup.

When we confirm that functionality is good and documentation is accurate, we shall release v.0.5.

Kind regards,

from echofish.

I will try to find some time next week to try it.

Thanks.

from echofish.

Closing the issue since we have no feedback. If you are still having issues feel free to open a new one.

from echofish.