zeek / zeekctl Goto Github PK
View Code? Open in Web Editor NEWTool for managing Zeek deployments.
Home Page: https://www.zeek.org
License: Other
Tool for managing Zeek deployments.
Home Page: https://www.zeek.org
License: Other
the quickstart guide says:
$PREFIX/logs/
current
However, when I have multiple loggers defined in my node.cfg:
[manager]
type=manager
host=localhost
[logger-1]
type=logger
host=localhost
[logger-2]
type=logger
host=localhost
[proxy-1]
type=proxy
host=localhost
[worker-1]
type=worker
host=localhost
interface=enp0s25
lb_procs=4
lb_method=custom
af_packet_fanout_id=1
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=67108864
I end up with logger-1
and logger-2
directories in the spool
directory, as expected. However, current
is still just a symlink to logger-1
. There's no access to logger-2
except through the spool
directory directly.
Not sure what the right thing to do here is... current-1
and current-2
, etc. symlinks? Or maybe just update the documentation.
On 0abed02, a configure on a system with only python3 installed results in:
$ ./configure
Build Directory : build
Source Directory: /Users/johanna/bro/master/auxil/zeekctl
-- The C compiler identification is AppleClang 12.0.0.12000032
-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found SWIG: /opt/local/bin/swig (found version "4.0.2")
-- Found PythonInterp: /usr/bin/python (found version "2.7.16")
-- Found PythonDev: /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
Found python version: 2.7.16
CMake Error at auxil/pysubnettree/CMakeLists.txt:16 (message):
Python 3.5 or greater is required.
-- Configuring incomplete, errors occurred!
When tucking file extraction onto the default local.zeek
, for example by adding
@load frameworks/files/extract-all-files
the extracted files are lost when shutting down Zeek nodes. The default extraction destination is spool/zeek/extracted_files/
, and zeekctl
's post-terminate
processor for cleaning up the spool first moves the entirety of spool/zeek
to a temp directory, then processes the logs in it (via archive-log
), and then discards the temp directory and any extracted files with it.
An easy fix could be for zeekctl
to adjust the FileExtract::prefix
to something that separates that location from its log processing. A bigger change would be to build out proper post-processing for finished files.
There's been lots of fixes since 1.9-2 that's shipping with Zeek 2.6.2, is there an intent to tag a new zeekctl release soon so these can be leveraged? Is the current 1.9-52 feature compatible with 2.6.2?
When running with multiple loggers and the ASCII writer, logs are rotated into the same destination directory without a discriminator, causing data loss.
The bigger topic is here #zeek/zeek#2728.
I am a newbie to Zeek and I am trying to build a Zeek cluster. But I encountered the following problems.
[logger-1]
No core file found.
Zeek 4.0.3
Linux 5.4.0-77-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
fatal error in /usr/local/zeek/share/zeek/base/frameworks/cluster/__load__.zeek, line 25: can't find cluster-layout
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=logger-1
==== .status
TERMINATED [fatal_error]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[manager]
No work dir found
[proxy-1]
No work dir found
[worker-1]
error running crash-diag for worker-1
Failed to establish ssh connection to host x.x.x.x
[ZeekControl] > Host key verification failed.
Host key verification failed.
Host key verification failed.
Host key verification failed.
Host key verification failed.
Host key verification faile
When I did not configure the cluster, I used sudo python3 zeekctl, and no error was reported. The above error occurred after modifying /etc/node.cfg.
==== No reporter.log
==== stderr.log
fatal error in /usr/local/zeek/share/zeek/base/frameworks/cluster/__load__.zeek, line 25: can't find cluster-layout
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=logger-1
==== .status
TERMINATED [fatal_error]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
These errors will also appear in non-cluster situations, and there are no files under /spool/zeek/. However, sometimes this problem does not arise. This confuses me. When I create these log files directly, I still get an error when running zeekctl.
Add a warning banner to zeekctl, such that zeekctl shipped with Zeek 5.0 will show a warning that a) there's an issue with the historic ZeekPort choice on Linux and b) that Zeek 5.2 will move this port and users are advised to make this change now.
Relates to #41
Hi there, I'm try to enable extract-all-file.zeek on worker , and it's worked.
when cluster start capture traffic , i can see the extracted file in /usr/local/zeek/spool/worker-1-1/extract_files on every worker.
but they are not aggregated to manager node,and will be cleaned when stop .
How can i aggregated those file in manager or logger node?
Built zeek on Fedora 39. When I run zeekctl help I get:
Traceback (most recent call last):
File "/usr/local/zeek/bin/zeekctl", line 814, in
sys.exit(main())
^^^^^^
File "/usr/local/zeek/bin/zeekctl", line 781, in main
loop = ZeekCtlCmdLoop(ZeekCtl, interactive, cmd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/zeek/bin/zeekctl", line 29, in init
self.zeekctl = zeekctl_class(ui=self)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/zeekctl.py", line 96, in init
self.setup()
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/zeekctl.py", line 111, in setup
self.config.initPostPlugins()
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/config.py", line 222, in initPostPlugins
self.nodestore = self._read_nodes()
^^^^^^^^^^^^^^^^^^
File "/usr/local/zeek/lib64/zeek/python/zeekctl/ZeekControl/config.py", line 389, in _read_nodes
config = configparser.SafeConfigParser()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'configparser' has no attribute 'SafeConfigParser'. Did you mean: 'RawConfigParser'?
The issue is SafeConfigParser has been removed on Python 3.12. There is a warning in Python 3.11 in
/usr/lib64/python3.11/configparser.py
class SafeConfigParser(ConfigParser):
"""ConfigParser alias for backwards compatibility purposes."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
warnings.warn(
"The SafeConfigParser class has been renamed to ConfigParser "
"in Python 3.2. This alias will be removed in Python 3.12."
" Use ConfigParser directly instead.",
DeprecationWarning, stacklevel=2
)
Paolo
Zeekctl currently always creates a link from share/zeekctl/scripts/zeekctl-config.sh
to ../../../spool/zeekctl-config.sh
. That link must be a relative link. Zeekctl will delete and recreate the link when it is an absolute link pointing to the same file.
For packaging purposes it would be great if zeekctl could just also accept an absolute link pointing to the same location.
More details:
When creating a binary package, relative links are forbidden in the package by virtually all distributions.
When the Zeek installation aims to be usable by a non-root-user, it is pretty inconvenient to make the share/zeekctl/scripts/zeekctl-config.sh
changeable by everyone (the directory needs to be writeable by the user who runs zeekctl to enable that - which seems not great).
The rotation.single-logger
test running at just the right time reports an .archive-log
process running, causing a baseline difference and a spurious error.
Failed here: https://cirrus-ci.com/task/5845006406123520?logs=test#L93
[#30] rotation.single-logger ... failed
% 'btest-diff logger_working_dir.out' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
.archive-log.running.107908.tmp
.cmdline
.env_vars
.pid
.startup
.status
== Diff ===============================
--- /dev/fd/63 2023-05-23 14:23:26.691588994 +0000
+++ /dev/fd/62 2023-05-23 14:23:26.691588994 +0000
@@ -1,3 +1,4 @@
+.archive-log.running.107908.tmp
.cmdline
.env_vars
.pid
=======================================
So, I saw this error when running zeekctl ( and zeekctl cron):
Error: must run zeekctl on same machine as the manager node. The manager node has IP address
127.0.1.1 and this machine has IP addresses: 128.3.x.y, fe80::3eec:efff:fe28:8c5a,
fe80::3eec:efff:fe28:8c5b, 127.0.0.1, ::1
Turns out this is some soft Linuxism where linux's netcfg
creates this entry in /etc/hosts
where
127.0.0.1 localhost
127.0.1.1 fqdn
This seems to have surfaced quite suddently and automatically on me.
While I changed the node.cfg to explicity use IP address instead of hostname as a workaround, I do see that /usr/local/zeek/lib/broctl/ZeekControl/config.py
has hardcoded 127.0.0.1, ::1
Would be useful if zeekctl
(aka config.py
) handles 127.0.0.0/8 instead
I needed to manually call archive-log due to an issue during our 2.6 -> 3.0 upgrade.
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log -h
archive-log: wrong usage: -h
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log
archive-log: wrong usage:
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log --help
archive-log: wrong usage: --help
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log help
archive-log: wrong usage: help
[zeek@bro-east1 ~]$ /usr/local/zeek/share/zeekctl/scripts/archive-log -?
archive-log: wrong usage: -?
Looks like the offending snippet is:
https://github.com/zeek/zeekctl/blob/master/bin/archive-log#L37~40
Seems way harsh to require 5 arguments, and not provide any kind of usage.
Zeek recently added a feature to statically build-in external plugins via zeek/zeek#1416. Unfortunately, zeekctl check
fails at startup with an following error if you have a plugin built in. This is using the af_packet plugin, for instance:
tim@nuc:~/builder$ zeekctl check
Hint: Run the zeekctl "deploy" command to get started.
zeek scripts failed.
fatal error in /home/tim/builder/install/share/zeek/builtin-plugins/__load__.zeek, line 3: can't find Zeek_AF_Packet/__load__.zeek
For reference, the default ZEEKPATH
now looks like .:/home/tim/builder/install/share/zeek:/home/tim/builder/install/share/zeek/policy:/home/tim/builder/install/share/zeek/site:/home/tim/builder/install/share/zeek/builtin-plugins:/home/tim/builder/install/share/zeek/builtin-plugins/Zeek_AF_Packet
.
An easy fix for the error displayed is to fix the set-zeek-path
script to also include ${policydir}/builtin-plugins
. This allows the check to pass.
With #48 and zeek/broker#21 popping up indicating out-dated baselines, it became clear that the zeekctl test suite isn't regularly being run.
This ticket tracks running it nightly against minimally the master branch. This might be implemented over in zeek/zeekctl.
Is this needed anymore? zeekctl was renamed a long long time ago, and I can't see any reason to keep installing a symlink for this. broctl.cfg
is still being installed as well.
After a fresh install of Zeek on FreeBSD system, I got this error when I ran this command:
zeekctl deploy
checking configurations ...
zeek scripts failed.
fatal error: can't find local.zeek
Zeek: 4.0.2
FreeBSD: 12.2
ZeekControl: 2.3.0
Apparently this file is not present under /usr/local/share/zeek/site (the local.zeek.sample file is present).
This didn't get handled in 4.0 and should probably be handled before 4.1 comes out.
The version of ps
that ships with Alpine doesn't have a -p
option. This means that broctl marks all bro processes as crashed.
There's a python-native option:
import os
def check_pid(pid):
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
return True
I think that this is reasonable, but I don't know if it is typical for the average Bro deployment to have Python on every node.
I am happy to put in a PR, but wanted thoughts on the approach first.
I think this is really an issue with broker, but I'm creating this issue first as it's the most user-visible.
Upgrading zeekctl to use Python3 gives the following error:
[zeek@zeek-box ~]$ zeekctl peerstatus
zeek-box-logger <error: Python bindings for Broker: No module named '_broker'>
zeek-box-manager <error: Python bindings for Broker: No module named '_broker'>
zeek-box-proxy <error: Python bindings for Broker: No module named '_broker'>
zeek-box-worker-1 <error: Python bindings for Broker: No module named '_broker'>
...
Hello
I was wondering about the most suitable way to achieve the following :
Thank you
I replaced the SafeConfigParser() with RawConfigParser() in the config.py and Zeekctl started working. Right now, I am wondering about the issues I might face for changing it in the future. I would love to see a feedback from your side about it. Thank You!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.