zeek / zeekctl Goto Github PK

Hi there, I'm try to enable extract-all-file.zeek on worker , and it's worked.

when cluster start capture traffic , i can see the extracted file in /usr/local/zeek/spool/worker-1-1/extract_files on every worker.
but they are not aggregated to manager node,and will be cleaned when stop .

How can i aggregated those file in manager or logger node?

When running with multiple loggers and the ASCII writer, logs are rotated into the same destination directory without a discriminator, causing data loss.

• Tag the logger node name onto the log name before running the log postprocessing to avoid the data-loss.
• Document

The bigger topic is here #zeek/zeek#2728.

I think this is really an issue with broker, but I'm creating this issue first as it's the most user-visible.

Upgrading zeekctl to use Python3 gives the following error:

[zeek@zeek-box ~]$ zeekctl peerstatus
zeek-box-logger   <error: Python bindings for Broker: No module named '_broker'>
zeek-box-manager   <error: Python bindings for Broker: No module named '_broker'>
zeek-box-proxy   <error: Python bindings for Broker: No module named '_broker'>
zeek-box-worker-1   <error: Python bindings for Broker: No module named '_broker'>
...

The rotation.single-logger test running at just the right time reports an .archive-log process running, causing a baseline difference and a spurious error.

Failed here: https://cirrus-ci.com/task/5845006406123520?logs=test#L93

[#30] rotation.single-logger ... failed
  % 'btest-diff logger_working_dir.out' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===============================
  .archive-log.running.107908.tmp
  .cmdline
  .env_vars
  .pid
  .startup
  .status
  == Diff ===============================
  --- /dev/fd/63	2023-05-23 14:23:26.691588994 +0000
  +++ /dev/fd/62	2023-05-23 14:23:26.691588994 +0000
  @@ -1,3 +1,4 @@
  +.archive-log.running.107908.tmp
   .cmdline
   .env_vars
   .pid
  =======================================

I am a newbie to Zeek and I am trying to build a Zeek cluster. But I encountered the following problems.

[logger-1]

No core file found.

Zeek 4.0.3
Linux 5.4.0-77-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log
fatal error in /usr/local/zeek/share/zeek/base/frameworks/cluster/__load__.zeek, line 25: can't find cluster-layout

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=logger-1

==== .status
TERMINATED [fatal_error]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[manager]
No work dir found

[proxy-1]
No work dir found

[worker-1]
error running crash-diag for worker-1
Failed to establish ssh connection to host x.x.x.x
[ZeekControl] > Host key verification failed.
Host key verification failed.
Host key verification failed.
Host key verification failed.
Host key verification failed.
Host key verification faile

When I did not configure the cluster, I used sudo python3 zeekctl, and no error was reported. The above error occurred after modifying /etc/node.cfg.

==== No reporter.log

==== stderr.log
fatal error in /usr/local/zeek/share/zeek/base/frameworks/cluster/__load__.zeek, line 25: can't find cluster-layout

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=logger-1

==== .status
TERMINATED [fatal_error]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

These errors will also appear in non-cluster situations, and there are no files under /spool/zeek/. However, sometimes this problem does not arise. This confuses me. When I create these log files directly, I still get an error when running zeekctl.

The version of ps that ships with Alpine doesn't have a -p option. This means that broctl marks all bro processes as crashed.

There's a python-native option:

import os

def check_pid(pid):
    try:
        os.kill(pid, 0)
    except ProcessLookupError:
        return False

    return True

I think that this is reasonable, but I don't know if it is typical for the average Bro deployment to have Python on every node.

I am happy to put in a PR, but wanted thoughts on the approach first.

After a fresh install of Zeek on FreeBSD system, I got this error when I ran this command:

zeekctl deploy

checking configurations ...
zeek scripts failed.
fatal error: can't find local.zeek

Zeek: 4.0.2
FreeBSD: 12.2
ZeekControl: 2.3.0

Apparently this file is not present under /usr/local/share/zeek/site (the local.zeek.sample file is present).

On 0abed02, a configure on a system with only python3 installed results in:

$ ./configure
Build Directory : build
Source Directory: /Users/johanna/bro/master/auxil/zeekctl
-- The C compiler identification is AppleClang 12.0.0.12000032
-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found SWIG: /opt/local/bin/swig (found version "4.0.2")
-- Found PythonInterp: /usr/bin/python (found version "2.7.16")
-- Found PythonDev: /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
Found python version: 2.7.16
CMake Error at auxil/pysubnettree/CMakeLists.txt:16 (message):
  Python 3.5 or greater is required.


-- Configuring incomplete, errors occurred!

There's been lots of fixes since 1.9-2 that's shipping with Zeek 2.6.2, is there an intent to tag a new zeekctl release soon so these can be leveraged? Is the current 1.9-52 feature compatible with 2.6.2?

With #48 and zeek/broker#21 popping up indicating out-dated baselines, it became clear that the zeekctl test suite isn't regularly being run.

This ticket tracks running it nightly against minimally the master branch. This might be implemented over in zeek/zeekctl.

I replaced the SafeConfigParser() with RawConfigParser() in the config.py and Zeekctl started working. Right now, I am wondering about the issues I might face for changing it in the future. I would love to see a feedback from your side about it. Thank You!

I needed to manually call archive-log due to an issue during our 2.6 -> 3.0 upgrade.

[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log -h
archive-log: wrong usage: -h
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log
archive-log: wrong usage:
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log --help
archive-log: wrong usage: --help
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log help
archive-log: wrong usage: help
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log -?
archive-log: wrong usage: -?

Looks like the offending snippet is:

https://github.com/zeek/zeekctl/blob/master/bin/archive-log#L37~40

Seems way harsh to require 5 arguments, and not provide any kind of usage.

Hello

I was wondering about the most suitable way to achieve the following :

1. Run zeekctl cron
2. If a zeek instance has crashed or stopped, run a custome script and restart the entire cluster.

Thank you

Built zeek on Fedora 39. When I run zeekctl help I get:

Traceback (most recent call last):
File "/usr/local/zeek/bin/zeekctl", line 814, in
sys.exit(main())
^^^^^^
File "/usr/local/zeek/bin/zeekctl", line 781, in main
loop = ZeekCtlCmdLoop(ZeekCtl, interactive, cmd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/zeek/bin/zeekctl", line 29, in init
self.zeekctl = zeekctl_class(ui=self)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/zeekctl.py", line 96, in init
self.setup()
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/zeekctl.py", line 111, in setup
self.config.initPostPlugins()
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/config.py", line 222, in initPostPlugins
self.nodestore = self._read_nodes()
^^^^^^^^^^^^^^^^^^
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/config.py", line 389, in _read_nodes
config = configparser.SafeConfigParser()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'configparser' has no attribute 'SafeConfigParser'. Did you mean: 'RawConfigParser'?

The issue is SafeConfigParser has been removed on Python 3.12. There is a warning in Python 3.11 in
/usr/lib64/python3.11/configparser.py

class SafeConfigParser(ConfigParser):
"""ConfigParser alias for backwards compatibility purposes."""

def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    warnings.warn(
        "The SafeConfigParser class has been renamed to ConfigParser "
        "in Python 3.2. This alias will be removed in Python 3.12."
        " Use ConfigParser directly instead.",
        DeprecationWarning, stacklevel=2
    )

Paolo

Zeek recently added a feature to statically build-in external plugins via zeek/zeek#1416. Unfortunately, zeekctl check fails at startup with an following error if you have a plugin built in. This is using the af_packet plugin, for instance:

tim@nuc:~/builder$ zeekctl check
Hint: Run the zeekctl "deploy" command to get started.
zeek scripts failed.
fatal error in /home/tim/builder/install/share/zeek/builtin-plugins/__load__.zeek, line 3: can't find Zeek_AF_Packet/__load__.zeek

For reference, the default ZEEKPATH now looks like .:/home/tim/builder/install/share/zeek:/home/tim/builder/install/share/zeek/policy:/home/tim/builder/install/share/zeek/site:/home/tim/builder/install/share/zeek/builtin-plugins:/home/tim/builder/install/share/zeek/builtin-plugins/Zeek_AF_Packet.

An easy fix for the error displayed is to fix the set-zeek-path script to also include ${policydir}/builtin-plugins. This allows the check to pass.

So, I saw this error when running zeekctl ( and zeekctl cron):

Error: must run zeekctl on same machine as the manager node. The manager node has IP address 
127.0.1.1 and this machine has IP addresses: 128.3.x.y, fe80::3eec:efff:fe28:8c5a, 
fe80::3eec:efff:fe28:8c5b, 127.0.0.1, ::1

Turns out this is some soft Linuxism where linux's netcfg creates this entry in /etc/hosts where

127.0.0.1 localhost
127.0.1.1 fqdn 

This seems to have surfaced quite suddently and automatically on me.

While I changed the node.cfg to explicity use IP address instead of hostname as a workaround, I do see that /usr/local/zeek/lib/broctl/ZeekControl/config.py has hardcoded 127.0.0.1, ::1

Would be useful if zeekctl (aka config.py) handles 127.0.0.0/8 instead

PS: https://serverfault.com/questions/363095/why-does-my-hostname-appear-with-the-address-127-0-1-1-rather-than-127-0-0-1-in

Zeekctl currently always creates a link from share/zeekctl/scripts/zeekctl-config.sh to ../../../spool/zeekctl-config.sh. That link must be a relative link. Zeekctl will delete and recreate the link when it is an absolute link pointing to the same file.

For packaging purposes it would be great if zeekctl could just also accept an absolute link pointing to the same location.

More details:

When creating a binary package, relative links are forbidden in the package by virtually all distributions.

When the Zeek installation aims to be usable by a non-root-user, it is pretty inconvenient to make the share/zeekctl/scripts/zeekctl-config.sh changeable by everyone (the directory needs to be writeable by the user who runs zeekctl to enable that - which seems not great).

This didn't get handled in 4.0 and should probably be handled before 4.1 comes out.

Add a warning banner to zeekctl, such that zeekctl shipped with Zeek 5.0 will show a warning that a) there's an issue with the historic ZeekPort choice on Linux and b) that Zeek 5.2 will move this port and users are advised to make this change now.

Relates to #41

the quickstart guide says:

• $PREFIX/logs/
  • As the name suggests it is the default logs directory where Zeek stores the rotated logs from the current directory:
  • current
    • It is a symlink to the spool directory that is defined in the zeekctl.cfg configuration file. It contains the active log files that Zeek currently writes to when running via ZeekControl.

However, when I have multiple loggers defined in my node.cfg:

[manager]
type=manager
host=localhost

[logger-1]
type=logger
host=localhost

[logger-2]
type=logger
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=enp0s25
lb_procs=4
lb_method=custom
af_packet_fanout_id=1
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=67108864

I end up with logger-1 and logger-2 directories in the spool directory, as expected. However, current is still just a symlink to logger-1. There's no access to logger-2 except through the spool directory directly.

Not sure what the right thing to do here is... current-1 and current-2, etc. symlinks? Or maybe just update the documentation.

When tucking file extraction onto the default local.zeek, for example by adding

@load frameworks/files/extract-all-files

the extracted files are lost when shutting down Zeek nodes. The default extraction destination is spool/zeek/extracted_files/, and zeekctl's post-terminate processor for cleaning up the spool first moves the entirety of spool/zeek to a temp directory, then processes the logs in it (via archive-log), and then discards the temp directory and any extracted files with it.

An easy fix could be for zeekctl to adjust the FileExtract::prefix to something that separates that location from its log processing. A bigger change would be to build out proper post-processing for finished files.