i have some problems on proxmox instalation debian11...

TZSP Replay crashing...
error log on syslog:

Mar 11 02:57:36 ids-suricata systemd[1]: Stopped TZSP Replay on dev tzsp0.
Mar 11 02:57:36 ids-suricata systemd[1]: Started TZSP Replay on dev tzsp0.
Mar 11 02:57:38 ids-suricata sh[3002]: Fatal Error: Error editing packet #2975: From edit_packet.c:fix_ipv4_checksums() line 74:
Mar 11 02:57:38 ids-suricata sh[3002]: Invalid packet: Expected IPv4 packet: got 0: pkt=2975
Mar 11 02:57:38 ids-suricata sh[3001]: error flushing via pcap_dump_flush
Mar 11 02:57:38 ids-suricata systemd[1]: [email protected]: Main process exited, code=exited, status=255/EXCEPTION
Mar 11 02:57:38 ids-suricata systemd[1]: [email protected]: Failed with result 'exit-code'.
Mar 11 02:57:41 ids-suricata systemd[1]: [email protected]: Scheduled restart job, restart counter is at 3.
Mar 11 02:57:41 ids-suricata systemd[1]: Stopped TZSP Replay on dev tzsp0.
Mar 11 02:57:41 ids-suricata systemd[1]: Started TZSP Replay on dev tzsp0.

any help?

Hi There, thanks for all your work with this. Works brilliantly

Having an issue with the Firewall address adding (auto blocking) stops at roughly the same time everyday.
All services are still running, selks still showing alerts, however not being communicated through to Mikrotik

Can confirm API user still showing logged in in Mikrotik log.

Only solution I can find is restarting Debian. Then starts working again instantly.

Things I noticed

• The timestamp comment in Mikrotik is grabbing the incorrect time. (13 hours behind) [I am +13GMT so assume this is related]
• The time of stoppage (according to the incorrect timestamp) is around 0100 - 0130 am (my time 2pm)

My mikrocataTZSP0.py config
mikrocataTZSP0.txt

systemd crashed every minute, I found this error:

Traceback (most recent call last):
  File "/usr/local/bin/mikrocataTZSP0.py", line 403, in <module>
    main()
  File "/usr/local/bin/mikrocataTZSP0.py", line 386, in main
    notifier.loop()
  File "/usr/local/lib/python3.9/dist-packages/pyinotify.py", line 1376, in loop
    self.process_events()
  File "/usr/local/lib/python3.9/dist-packages/pyinotify.py", line 1275, in process_events
    self._default_proc_fun(revent)
  File "/usr/local/lib/python3.9/dist-packages/pyinotify.py", line 910, in __call__
    return _ProcessEvent.__call__(self, event)
  File "/usr/local/lib/python3.9/dist-packages/pyinotify.py", line 630, in __call__
    return meth(event)
  File "/usr/local/bin/mikrocataTZSP0.py", line 77, in process_IN_MODIFY
    add_to_tik(read_json(FILEPATH))
  File "/usr/local/bin/mikrocataTZSP0.py", line 114, in read_json
    alerts = [ujson.loads(line) for line in f.readlines()]
  File "/usr/local/bin/mikrocataTZSP0.py", line 114, in <listcomp>
    alerts = [ujson.loads(line) for line in f.readlines()]
ujson.JSONDecodeError: Unmatched '"' when decoding 'string'

I rewrote the read_json function and now the script works stably without errors:

 def read_json(fpath):
    global last_pos

    try:
        with open(fpath, "r") as f:
            f.seek(last_pos)
            alerts = []

            for line in f.readlines():
                try:
                    alerts.append(ujson.loads(line))
                except ujson.JSONDecodeError as e:
                    print(f"JSON decoding error: {e}\nLine: {line}")
                    
            last_pos = f.tell()
            return alerts

    except FileNotFoundError:
        print(f"[Mikrocata] File: {fpath} not found. Retrying in 10 seconds..")
        sleep(10)

The easyinstall.sh script will work on Ubuntu 20.04, but the command for docker compose is "docker compose up -d" instead of "docker-compose up -d".

Maybe add a flavour detection to the script?

Hi! In the /var/lib/docker/volumes/selks_scirius-data/_data/git-sources/1/rules/classification.config file in line 66, I changed it to 2 instead of 1, updated the set of rules and it works. Resets the classification for the next day. Tell me how to fix it?

kindly share your paypal, i like your project working as expected, i would like to support to keep it UP

Hi! Help me figure it out, Suricata does not send data to 3 Mikrotik.

I did everything according to the instructions. I use Debian 12.5.

The packets arrive on 3 virtual interfaces: tzsp0, tzsp1, tzsp2.

I stopped 3 services: mikrocataTZSP0.service, mikrocataTZSP1.service, mikrocataTZSP2.service.

I started it:python3 /usr/local/bin/mikrocataTZSP0.py

python3 /usr/local/bin/mikrocataTZSP1.py

python3 /usr/local/bin/mikrocataTZSP2.py

Everything works correctly only with the ET rule, which is integrated into the repository.

Any new rullset does not work. Despite many attempts to update

Perhaps it is a matter of a new SELKS update

Hello,

I've been playing around with your project for the past few weeks and I'd like to start by expressing my appreciation for the work you've been doing. It's been a smooth experience until recently when I encountered a persistent issue.

Starting from March 1st, I've been facing a Socket timeout error that's hindering the connectivity between Suricata and the Mikrotik API. The systemd logs pinpoint the start of the issue to exactly 14:30,. However, merely 12 seconds later, a timeout error.

Any idea what could cause this?

Mar 01 14:30:00 mikrocata systemd[1]: Started Suricata to Mikrotik API in Python.
Mar 01 14:30:12 mikrocata python3[2431173]: [Mikrocata] Socket timeout: timed out.

Thanks for advance.

Hello!
Thank you very much for your efforts. Please tell me. My firewall is managed by MikroTik.
SELKS is installed on Debian 11 listening to traffic on standard network ports (enp5s0, enp6s0). Traffic going to MikroTik is mirrored in parallel to SELKS using a smart switch. How can I use Mikrocata2 to manage my MikroTik firewall? I need to manage the MikroTik firewall only when Suricata rules are triggered.
I would appreciate your help.

Hi,

I have been running this perfectly in prod for last few months. Had an issue within my vm ESXI which meant I needed to rebuild from scratch. (Backups were also affected)

Been running for around a week to get my rules back in order, connected today to my Mikrotik and noticed in Mikrotik logs the API user was disconnecting / reconnecting

Traced it back to microcata.py service crashing when it is trying to log firewall rule

Logs during the crash below

[Mikrocata] Connected to Mikrotik Traceback (most recent call last): File "/usr/local/bin/mikrocataTZSP0.py", line 414, in <module> main() File "/usr/local/bin/mikrocataTZSP0.py", line 398, in main notifier.loop() File "/usr/lib/python3/dist-packages/pyinotify.py", line 1376, in loop self.process_events() File "/usr/lib/python3/dist-packages/pyinotify.py", line 1275, in process_events self._default_proc_fun(revent) File "/usr/lib/python3/dist-packages/pyinotify.py", line 910, in __call__ return _ProcessEvent.__call__(self, event) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pyinotify.py", line 630, in __call__ return meth(event) ^^^^^^^^^^^ File "/usr/local/bin/mikrocataTZSP0.py", line 78, in process_IN_MODIFY add_to_tik(read_json(FILEPATH)) File "/usr/local/bin/mikrocataTZSP0.py", line 159, in add_to_tik if event["src_ip"].startswith(WHITELIST_IPS): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: tuple for startswith must only contain str, not tuple

Hello

I see in the router that new IP addresses are added to the letter, but the message does not come in Telegram. Where can I see the log of sending a message in Telegram?

Thanks

amp=2023-02-27T03:16:43+00:00 tags=["error","elasticsearch-service"] pid=7
Error: Timeout: it took more than 1200000ms
at Timeout._onTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:54:20)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7) | type=log @timestamp=2023-02-27T03:17:26+00:00 tags=["error","plugins","ruleRegistry"] pid=7
Error: Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms
at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:63:13)
at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:77:5) | type=log @timestamp=2023-02-27T03:17:26+00:00 tags=["error","plugins","ruleRegistry"] pid=7
Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.18.0.4:9200 | type=log @timestamp=2023-02-27T03:17:42+00:00 tags=["error","elasticsearch-service"] pid=7
Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND elasticsearch | type=log @timestamp=2023-02-27T03:17:44+00:00 tags=["error","elasticsearch-service"] pid=7
Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.18.0.4:9200 | type=log @timestamp=2023-02-27T03:18:45+00:00 tags=["error","elasticsearch-service"] pid=7
Unable to retrieve version information from Elasticsearch nodes. getaddrinfo ENOTFOUND elasticsearch | type=log @timestamp=2023-02-27T03:18:49+00:00 tags=["error","elasticsearch-service"] pid=7

Hi,
I was wondering why the service "[email protected]" gets an error after 10 or 20 seconds. This is what I can see on the "cli".

systemctl status [email protected]
● [email protected] - TZSP Replay capture on dev tzsp0
     Loaded: loaded (/etc/systemd/system/[email protected]; enabled; preset: enabled)
     Active: active (running) since Tue 2024-07-16 18:39:15 CEST; 22s ago
   Main PID: 489493 (sh)
      Tasks: 3 (limit: 9482)
     Memory: 1.9M
        CPU: 202ms
     CGroup: /system.slice/system-TZSPreplay37008.slice/[email protected]
             ├─489493 /bin/sh -c "/usr/local/bin/tzsp2pcap -p 37008 -f | /usr/local/bin/tcpreplay-edit --topspeed --mtu=\$(cat /sys/class/net/tzsp0/mtu) --mtu-trun>
             ├─489495 /usr/local/bin/tzsp2pcap -p 37008 -f
             └─489496 /usr/local/bin/tcpreplay-edit --topspeed --mtu=2000 --mtu-trunc -i tzsp0 -

Jul 16 18:39:15 VMidsips systemd[1]: Started [email protected] - TZSP Replay capture on dev tzsp0.
Jul 16 18:39:30 VMidsips sh[489496]: Warning: Unable to process unsupported DLT type: Ethernet (0x1)
Jul 16 18:39:30 VMidsips sh[489496]: Warning: Unable to process unsupported DLT type: Ethernet (0x1)

I installed an debian as an vm with proxmox.

uname -a
Linux VMidsips 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

Evrything else is working. Suricata connects to the mikroTik and uses the function to block traffic. Therefor the instructions how to install this docker-enviroment works. This VM has got 6 CPU and 8GB RAM and 60GB filestorage.

Where can I find help or a solution to the problem?

thnx 4 your help.

树莓派4B 8G，应该是可以运行的。但遗憾的是arkime和scirius没有arm64架构

Here is some code that fixes the API timeout issue when trying to connect with self-signed certificates on the Mikrotik.

I have used this code and it works as expected.

#!/usr/bin/env python3

import ssl
import os
import socket
import re
from time import sleep
from datetime import datetime as dt
import pyinotify
import ujson
import json
import librouteros
from librouteros import connect
from librouteros.query import Key
import requests

# ... (earlier parts of the script remain unchanged)

def connect_to_tik():
    global api
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE  # Add this line
    ctx.set_ciphers('DEFAULT@SECLEVEL=1')  # Modified this line

    while True:
        try:
            api = connect(username=USERNAME, password=PASSWORD, host=ROUTER_IP,
                          ssl_wrapper=ctx.wrap_socket, port=PORT)
            print(f"[Mikrocata] Connected to Mikrotik")
            break

        except librouteros.exceptions.TrapError as e:
            if "invalid user name or password" in str(e):
                print("[Mikrocata] Invalid username or password.")
                sleep(10)
                continue

            raise

        except socket.timeout as e:
            print(f"[Mikrocata] Socket timeout: {str(e)}.")
            sleep(30)
            continue

        except ConnectionRefusedError:
            print("[Mikrocata] Connection refused. (api-ssl disabled in router?)")
            sleep(10)
            continue

        except OSError as e:
            if e.errno == 113:
                print("[Mikrocata] No route to host. Retrying in 10 seconds..")
                sleep(10)
                continue

            if e.errno == 101:
                print("[Mikrocata] Network is unreachable. Retrying in 10 seconds..")
                sleep(10)
                continue

            raise

        except ssl.SSLError as e:
            print(f"[Mikrocata] SSL Error: {str(e)}. Retrying in 10 seconds..")
            sleep(10)
            continue

# ... (rest of the script remains unchanged)

Hello,
first I would like to thank you for this easy and working solution of your script!

Do you think that will be possible to modified your code for filtering each alert output based on severity?

For example I would like to send to the MikroTik Firewall only IPs with severity:1 (red ones). And ignore the other two rules.

Thank you for your time and answer... 💯
Regards...

I also made some changes in the add_to_tick function to suppress the error 'key error'

def add_to_tik(alerts):
    global last_pos
    global api
    _address = Key("address")
    _id = Key(".id")
    _list = Key("list")
    address_list = api.path("/ip/firewall/address-list")
    resources = api.path("system/resource")
        for event in {item['src_ip']: item for item in alerts}.values():
            if 'alert' not in event:
                print("Event does not contain a key 'alert':", event)
                continue
            signature_id = event['alert'].get('signature_id', 'unknown')
            signature = event['alert'].get('signature', 'unknown')
            timestamp = event.get("timestamp", "unknown")
            src_ip = event.get("src_ip", "unknown")
            dest_ip = event.get("dest_ip", "unknown")
            src_port = event.get("src_port", "unknown")
            proto = event.get("proto", "unknown")
            wanted_ip, wanted_port = event["dest_ip"], event.get("src_port")
            else:
                wanted_ip, wanted_port = event["src_ip"], event.get("dest_port")
            try:
                address_list.add(list=BLOCK_LIST_NAME,
                                 address=wanted_ip,
                                 comment=f"""[{event['alert']['gid']}:{
                                 event['alert']['signature_id']}] {
                                 event['alert']['signature']} ::: Port: {
                                 wanted_port}/{
                                 event['proto']} ::: timestamp: {
                                 timestamp}""",
                                 timeout=TIMEOUT)

            except librouteros.exceptions.TrapError as e:
                if "failure: already have such entry" in str(e):
                    for row in address_list.select(_id, _list, _address).where(
                            _address == wanted_ip,
                            _list == BLOCK_LIST_NAME):
                        address_list.remove(row[".id"])

                    address_list.add(list=BLOCK_LIST_NAME,
                                     address=wanted_ip,
                                     comment=f"""[{event['alert']['gid']}:{
                                     event['alert']['signature_id']}] {
                                     event['alert']['signature']} ::: Port: {
                                     wanted_port}/{
                                     event['proto']} ::: timestamp: {
                                     timestamp}""",
                                     timeout=TIMEOUT)

                else:
                    raise

            except socket.timeout:
                connect_to_tik()