Using logstash queries to enrich log entries about redelk HOT 3 CLOSED

I guess you mean the python (enrich.py) script instead of 'ruby script'? Ruby is only used for inserting links to URLs visible in Kibana.

The approach you mention was our first thought as well and we have gone down this route in our very first versions of RedELK. However we experienced a series of issues, mainly:

1. if the first "metadata" line of CS is not yet available in ES, the lookup will fail. In our testing this happened too much. Afaik this wasnt a resource issue, but an issue when the filebeat-logstash link was down and they were playing catchup.
2. There is no way of historically modifying things. If the lookup failed, the grok parsing failed and will keep as failed forever. This really comes to play in the 2nd point of your post, the IP lists. At least in our operations we have increasing knowledge of the different sets of IP addresses. We want those historically changed as well. IP is just an example. We can use enrich.py on other info as well that we like to add during the ops (including other C2 frameworks). It is a great template for historically modifying data.

Now, it has been a time since we looked at his (ES version 5.x), and perhaps things have changed. A solution for the first issue could as well be to start using a setup with Kafka and KSQL. But tbh it does seem a bit as overkill and complex.
Yet at the same time I can imagine that the current enrich.py - although it works - might seem overkill and overly complex as well. It might not give the same level of flexibility we really enjoy if you aren't comfortable with the code...?

Happy with your thoughts and overall happy with you thinking along with the project! Im open for discussion, just wanted to let you know we have looked at this some time ago.

from redelk.

Happy to leave it open for some other time. But if it isn't relevant anymore, I would like to close this one.

from redelk.

I think as per our discussion last time we can close this one. Using the enrich.py instead of logstash queries provides more flexibility (like enriching historical data).

from redelk.