mosajjal / dnsmonster Goto Github PK

Would it be possible to be able to limit the size of the output json files by packet count?

As you'd be aware pcap to json conversion balloons the size of the files, for us it's taking a gzip compressed pcap from 1GB to a JSON file of 26GB.

Would be great if we could split that out with the following example:

Input:
example.pcap

Output:
example_001.json
example_002.json
etc etc

Also helps with downstream horizontal scaling as the files are processed off S3, so smaller but more numerous files allows more simultanteous workers to process the output.

Hello,
First, I would like to thank you for this excellent and useful project!
Regarding Clickhouse output, I am running a local stack and while monitoring my main interface, I cannot find CNAME queries in DNS_LOG table (Types are either 1 or 28 which are mapped to A and AAAA respectively). Is this an issue with my configuration or is this by default?

Best,

First of all thank you for this amazing dnsmonster !!!
We have a cluster of three clickhouse nodes and our dns data is going through dnsmonster. But in the configuration of dnsmonster only one IP address of the clickhouse node is mentioned. The problem is that if our mentioned clickhouse node goes down, will our data replicate to the other cluster node?

Kindly Suggest !!!

Thanks !!!

I made a small change that leverages builtin driver's support. Most likely I'll make a change to the way the Address is provided so we can have different credentials and TLS support for each address later on. Please let me know if the latest commit solves your issue. Happy to re-open if it doesn't

Originally posted by @mosajjal in #27 (comment)

If we have multiple servers sending data over dnstap to dnsmonster it would be good to have an option to use the dnstap identity field as the server name which gets recorded in the logs.

Hi,,
Can dnsmonster handle 5,000,000 QPS? We have a passive server about receive DNS traffic, and we want analyze the traffic data by dnsmonster, I'm not sure if it's a best way？
bellow is my server network info：

Unsure if this is intentional but the Prometheus endpoint command-line option and environment variable is

--metricPromethusEndpoint

and not

--metricPrometheusEndpoint

Thanks for writing a really useful tool for pcap parsing into dns json!

I just have a question, I'm running the command as such below:

$ dnsmonster --pcapfile="output.pcap" --fileoutputpath=dns.json --fileoutputformat=json --fileoutputtype=1
INFO[2022-11-04T15:44:41Z] Creating the dispatch Channel
INFO[2022-11-04T15:44:41Z] Creating File Output Channel
INFO[2022-11-04T15:44:41Z] Using File: output.pcap

INFO[2022-11-04T15:44:41Z] skipping skipDomains refresh since it's not provided
INFO[2022-11-04T15:44:41Z] skipping allowDomains refresh since it's not provided
WARN[2022-11-04T15:44:41Z] BPF Filter is not supported in offline mode.
INFO[2022-11-04T15:44:41Z] Reading off Pcap file
INFO[2022-11-04T15:44:41Z] Creating handler #0
INFO[2022-11-04T15:44:41Z] Creating handler #1
2022-11-04T15:44:51Z metrics: {"fileSentToOutput":{"count":136405},"fileSkipped":{"count":0},"packetLossPercent":{"value":0},"packetsCaptured":{"value":0},"packetsDropped":{"value":0},"packetsDuplicate":{"count":0},"packetsOverRatio":{"count":0}}
2022-11-04T15:45:01Z metrics: {"fileSentToOutput":{"count":136405},"fileSkipped":{"count":0},"packetLossPercent":{"value":0},"packetsCaptured":{"value":0},"packetsDropped":{"value":0},"packetsDuplicate":{"count":0},"packetsOverRatio":{"count":0}}
...

But this never ends? According to top/iotop the process has finished writing and I can confirm the output json file seems to have stopped writing - but dnsmonster never terminates back to the shell.

Is this expected behaviour?

Hello !!
hope you are doing well.
how to make dns-monster listen multiple ports ???

It seems like the default config is just batching clickhouse output and only writing when the batch is full, which is a bit confusing - changing to using 1s flush interval makes the flushes work correctly in low-quantity testing.

May I suggest that you have a flush algorithm that flushes EITHER when flush interval OR batch size is reached as that is the standard way to do things? And that flush interval is set to eg 5s as default?

Is it possible to extend dnsmonster to store TCP RTT values during initial handshake (difference between first and third packet)?

Paper behind on how useful TCP RTT data even if its limited compare to UDP dns
https://ant.isi.edu/~johnh/PAPERS/Moura20a.pdf

Carrying on from the TCP race condition issue.

Seems there's some discrepancies in the packet counting when loading a pcap file.

Please give me suggestion

I noticed that the number of DNS packets stored in my local clickhouse instance were always multiples of the clickhousebatchsize:
When using dnsmonster to process pcap files and clickhousebatchsize set to non-zero, the clickhouse output will not send all results to clickhouse.

Scenario:

• clickhousebatchsize = 100000 (default)
• output to clickhouse
• pcaps with number of DNS packets which is not multiple of the batch size, e.g. 1234567

Result:
dnsmonster sends info for only 1200000 packets to clickhouse, in batches of 100000.
Missing the 34567 packets as the batch size was not reached.

The code section responsible is

I did not see anything that makes sure the remainder is being sent to clickhouse when the end of the pcap file has been reached.

First of all I would like to say a big thank you for making this.

Everything gets installed fine, even getting the logs on clickhouse container when checked clickhouse-client, So that works brilliantly. However, while looking at the datasource in grafana. The local ch datasource doesn't load, tried adding new then noticed following error at the top

Tried changing the flag "allow_loading_unsigned_plugins" too, but the error is not going away. Clickhouse is not available in the list of datasources.

In the README.MD in the Configuration File section, the command line is listed as

sudo dnsmonster -config=dnsmonster.ini

It should be

sudo dnsmonster --config=dnsmonster.ini

Sorry to raise this but I'm failing to compile the main branch.

Error:

go build -o dnsmonster .
# github.com/mosajjal/dnsmonster/capture
capture/livecap_bsd.go:69:26: h.sniffer.Stats undefined (type bsdbpf.BPFSniffer has no field or method Stats)

Mac OS 12.6
Go 1.19.3 darwin/arm64

I noticed that the scheme appears to have changed and I tried to apply that to the replicated example, but it is not working as expected.

Hello,

It seems the filtering/allow logic is broken for the Splunk HEC Output as once the splunkOutputType is increased above 1, all domains get skipped, no matter what is in/not in the allow/skip files:
{"level":"info","msg":"output: {Name:splunk SentToOutput:0 Skipped:99441}","time":"2021-07-13T15:03:29+10:00"}
{"level":"info","msg":"{PacketsGot:99485 PacketsLost:0 PacketLossPercent:0}","time":"2021-07-13T15:03:29+10:00"}

Config file:
useAfpacket=true
devName=myerspan
splunkOutputType=3
skipDomainsFile=/app/dnsmonster/filterDomains.csv
splunkOutputEndpoint=:8088
splunkOutputToken=
skipTlsVerification=true
splunkOutputIndex=
splunkOutputSource=
splunkOutputSourceType=

filterDomains:
empty

Are you able to please advise if something is wrong with the config or if this bug has been fixed in commits past the 8.4.0 release?

Thanks,
Lachlan

Hi ,
We want central logging by Extending DNSMonnster's capability . As in Case of SSL/TLS web traffic we can decrypt a encrypted SSL/TLS session if we have access of server private key this is a very standard process in wireshark

https://my.f5.com/manage/s/article/K19310681

can we work on DNS Monster to have this feature to add optional priavte key in customised setup so that we can have decrypted DoT / DoH logs in table

Hi ..
we are observing a issue in monster while logging data to click-house from monster.
we have two standalone VMs in same location and configured monster with latest version.
but in one VMs data are being logged less than second VMs.
Supoose we have 100 packets of data then one VMs loogged 80 packets and second VMs logged 30 Packets.
I suppose that both VMs should logged full 100 of packets without any loss.

Is it possible to pipe pcap data to dnsmonster? Sometimes a network stream needs first to be decapsulated.

Something like pcap.OpenOfflineFile(os.Stdin)?

Thanks for your project, it helped me a lot. I found that the Kafka SASL_PLAINTEXT protocol is not currently supported. Could please support this protocol?

It would be nice to have higher precision timestamps for the package's time in clickhouse.

The current DNS_Log table:

Am not sure why IndexTime is higher precision, but PacketTime is just seconds.

The PostgreSQL output uses timestamp for both

Maybe the PacketTime column in clickhouse's DNS_LOG could be changed to the higher-resolution https://clickhouse.com/docs/en/sql-reference/data-types/datetime64 as well?

Just changing the data type for the column seems to work.

I've noticed that on a number of the CH tables you are including timestamp in the order by. Rather than doing this you should probably have a truncated timestamp such as by minute (or at least truncated to per-second) otherwise there's not much point in the MV's compared to just sampling from the raw table itself.

Additionally there are a number of times when you sum() a value such as DoBit which is a UInt8 in the primary table. It would be better to cast those to UInt64 and then sum that to avoid overflows.

Hi ,
Kindly look into this !!!
Dnsmonster Service.
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="Starting DNStap capture"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="listening on DNStap socket unix:///var/cache/bind/dnstap1.sock"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="socket exists, will try to overwrite the socket"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="Creating handler #0"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="Creating handler #1"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="Creating the dispatch Channel"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="Creating Clickhouse Output Channel"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="skipping skipDomains refresh since it's not provided"
Jun 22 14:12:50 two dnsmonster[19067]: time="2023-06-22T14:12:50+05:30" level=info msg="skipping allowDomains refresh since it's not provided"
Jun 22 14:13:00 two dnsmonster[19067]: time="2023-06-22T14:13:00+05:30" level=warning msg="failed to convert metrics to JSON."
Jun 22 14:13:00 two dnsmonster[19067]: 2023-06-22T14:13:00+05:30 metrics:
Jun 22 14:13:00 two dnsmonster[19067]: time="2023-06-22T14:13:00+05:30" level=info msg="ipv4 flushed: 0, closed: 0"
Jun 22 14:13:00 two dnsmonster[19067]: time="2023-06-22T14:13:00+05:30" level=info msg="ipv6 flushed: 0, closed: 0"
Jun 22 14:13:10 two dnsmonster[19067]: time="2023-06-22T14:13:10+05:30" level=warning msg="failed to convert metrics to JSON."
Jun 22 14:13:10 two dnsmonster[19067]: 2023-06-22T14:13:10+05:30 metrics:
Jun 22 14:13:10 two dnsmonster[19067]: time="2023-06-22T14:13:10+05:30" level=info msg="ipv4 flushed: 0, closed: 0"
Jun 22 14:13:10 two dnsmonster[19067]: time="2023-06-22T14:13:10+05:30" level=info msg="ipv6 flushed: 0, closed: 0"
Jun 22 14:13:20 two dnsmonster[19067]: time="2023-06-22T14:13:20+05:30" level=warning msg="failed to convert metrics to JSON."
Jun 22 14:13:20 two dnsmonster[19067]: 2023-06-22T14:13:20+05:30 metrics:
Jun 22 14:13:20 two dnsmonster[19067]: time="2023-06-22T14:13:20+05:30" level=info msg="ipv4 flushed: 0, closed: 0"
Jun 22 14:13:20 two dnsmonster[19067]: time="2023-06-22T14:13:20+05:30" level=info msg="ipv6 flushed: 0, closed: 0"
Jun 22 14:13:30 two dnsmonster[19067]: time="2023-06-22T14:13:30+05:30" level=warning msg="failed to convert metrics to JSON."
Jun 22 14:13:30 two dnsmonster[19067]: 2023-06-22T14:13:30+05:30 metrics:
Jun 22 14:13:30 two dnsmonster[19067]: time="2023-06-22T14:13:30+05:30" level=info msg="ipv4 flushed: 0, closed: 0"
Jun 22 14:13:30 two dnsmonster[19067]: time="2023-06-22T14:13:30+05:30" level=info msg="ipv6 flushed: 0, closed: 0"

Fairly simple question, would it be possible to have gzip compression options on the file output option?

Saves me having to use a file watcher on a directory to collect files and compress them prior to AWS S3 upload.

`dnsmonster[8323]: time="2023-01-24T12:38:07+05:30" level=warning msg="Error while executing batch: code: 43, message: Illegal type DateTime64(3) of argument for aggregate function sumWithOverflow: while pushing to view default.DNS_PROTOCOL (f20ac7c5-9b15-4d2f-935f-7fb13614c12f)"

Kindly look into it !!!
Thanks !!!

In "IPv6 Packet Destination Top 20 Prefix" panel, the IPv6 addresses are incorrect.

e.g. in my panel ,it is showing random IPs.

In csv.go page, the code for converting IPv6 address to decimal is --

SrcIP = binary.BigEndian.Uint64(d.SrcIP[:8]) //limitation of clickhouse-go doesn't let us go more than 64 bits for ipv6 at the moment DstIP = binary.BigEndian.Uint64(d.DstIP[:8])

As per my understanding from above comment, clickhouse is not allowing more than 64 bits for a variable. Is there a way to show correct data on the panel ?

When I use grafana panel.json (https://github.com/mosajjal/dnsmonster/blob/main/grafana/panel.json), I found 2 cahrt that grafana can't draw:
,
Then I found that two SQL statements encountered errors while executing.

SELECT 0, groupArray((IP, total)) FROM (SELECT IPv6NumToString(toFixedString(SrcIP, 16)) AS IP,sum(c) as total FROM DNS_SRCIP_MASK PREWHERE IPVersion=4 WHERE $timeFilter      GROUP BY SrcIP order by SrcIP desc limit 20);
SELECT 0, groupArray((IP, total)) FROM (SELECT IPv6NumToString(toFixedString(SrcIP, 16)) AS IP,sum(c) as total FROM DNS_SRCIP_MASK PREWHERE IPVersion=6 WHERE $timeFilter     GROUP BY SrcIP order by SrcIP desc limit 20)

After converting to regular SQL statements and executing them in ClickHouse, the following error is displayed below:

SELECT
    0,
    groupArray((IP, total))
FROM
(
    SELECT
        IPv6NumToString(toFixedString(SrcIP, 16)) AS IP,
        sum(c) AS total
    FROM DNS_SRCIP_MASK
    PREWHERE IPVersion = 4
    WHERE (DnsDate >= toDate(1697869430)) AND (DnsDate <= toDate(1697880230)) AND (timestamp >= toDateTime(1697869430)) AND (timestamp <= toDateTime(1697880230))
    GROUP BY SrcIP
    ORDER BY SrcIP DESC
    LIMIT 20

Query id: 82ad98fe-6377-4980-9325-a9ca63682f27


0 rows in set. Elapsed: 0.002 sec.

Received exception from server (version 23.3.14):
Code: 48. DB::Exception: Received from localhost:9000. DB::Exception: toFixedString is only implemented for types String and FixedString: While processing IPv6NumToString(toFixedString(SrcIP, 16)) AS IP, sum(c) AS total. (NOT_IMPLEMENTED)
)

7a3c86218e97 :) select * from DNS_SRCIP_MASK

SELECT *
FROM DNS_SRCIP_MASK

Query id: a0453307-81e5-4347-a840-e5cd9381946b

┌────DnsDate─┬───────────timestamp─┬─Server──┬─IPVersion─┬─SrcIP─────────────────┬───c─┐
│ 2023-10-21 │ 2023-10-21 06:51:18 │ default │         4 │ ::ffff:172.23.160.1   │ 136 │
│ 2023-10-21 │ 2023-10-21 06:51:18 │ default │         4 │ ::ffff:172.23.162.110 │ 126 │
└────────────┴─────────────────────┴─────────┴───────────┴───────────────────────┴─────┘
┌────DnsDate─┬───────────timestamp─┬─Server──┬─IPVersion─┬─SrcIP───────────────┬─c─┐
│ 2023-10-21 │ 2023-10-21 09:18:32 │ default │         4 │ ::ffff:172.23.160.1 │ 1 │
└────────────┴─────────────────────┴─────────┴───────────┴─────────────────────┴───┘
┌────DnsDate─┬───────────timestamp─┬─Server──┬─IPVersion─┬─SrcIP─────────────────┬─c─┐
│ 2023-10-21 │ 2023-10-21 09:18:32 │ default │         4 │ ::ffff:172.23.162.110 │ 1 │
└────────────┴─────────────────────┴─────────┴───────────┴───────────────────────┴───┘

Hi !!

Hope You are doing Well.
I downloaded the latest dnsmonster(v0.9.5) binary from the release section . but dnsmonser not sending packets from live interface to clickhouse.
The error is given Below :--
[4515]: time="2022-10-07T18:11:53+05:30" level=warning msg="Error while executing batch: clickhouse [Append]: clickhouse: expected 18 arguments, got 17"

but in your previous version binary like (v0.9.2 & v0.9.3) is working fine ...

kindly check and do the needful ....

Thanks !!!!

https://datatracker.ietf.org/doc/draft-dulaunoy-dnsop-passive-dns-cof/

looks like a cool idea to implement