deepfence / packetstreamer Goto Github PK

View Code? Open in Web Editor NEW

1.9K 13.0 252.0 2.11 MB

:star: :star: Distributed tcpdump for cloud native environments :star: :star:

License: Apache License 2.0

Makefile 1.14% Go 93.84% Dockerfile 1.08% Smarty 3.94%

soc network-analysis tcpdump-like packet-capture packet-sniffer observability security-tools snort zeek suricata

packetstreamer's People

Contributors

Stargazers

Watchers

Forkers

mandrwg jimba86 kjm0001 mflu huangweiboy2 omaramin17 ozzykampha isiaon killvxk eternalerrors snk0752 bayanijulian blue-infosec auspex-labs crt-fork dio r8ygun derekhjray whit3rabbit matosdiego leszekgrzegorek cheahengsoon sec-js 0xsojalsec 5l1v3r1 mathusan-vadivel dekoder wanglibin1314 ogarrett vadorovsky rokoman dabaosun abin2024 saksharora prashant7454 jigarvyasidea abbansal235 prayag2002 pradhuman26 harshit1agarwal kushpatel2106 gulkhan799 tushar99coder ayushnirmal16 priyatiwari72 rohit1443 manojkumarmk123 shubhgarg15 shoubik08 ayushpant007 sukhvendra-tak pradeep9057 ritikbhati22 harshit0421 dishant7792 y-47 punitydv98 siddarthsuhane vsharma12345 prajapatdhawal prernadadhich19 praksh1924 sourav092002 iq-scm vaskata84 unlockunl feichi22365 manguala ratnokustiawan nawabsahab16 harrywang29 vedantkolhe30 shoaibxaif amanbairagi30 shashwat79802 knighthawk-leo miarkable shanks2121 ishukrpathak kavyaswarnkar harshnow sitanshulall akashtiwari2003 manas234543 shrutiiihere ranjit7858 theayushgaur txtgrey yash-sakre bhumika987 prabhakarjha04 prateekmahant6 devansh0331 raghavbharani mohdadnan786 sanskritiharmukh jayashtripathy vinayak27-raw carrycooldude amit7nayak

packetstreamer's Issues

Compile/build for Windows.

Simply add support for use with Windows. Or how we can compile for Windows.

Thank you.

Packet capture not working on kubernetes

I Installed the default helm installation of packet streamer as daemonset on kubernetes but I am not able to capture any traffic. The sensor shows the traffic capture count, but the receiver side the the logs shows 0B and also the size of the /tmp/dump file never increase.
If i enable to save packets at the sensor side, again I don't see the size of the /tmp/dump file increasing.

To Reproduce
kind create cluster --config config.yaml

three node (two workers) cluster config

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cluster1
nodes:

role: control-plane
image: kindest/node:v1.23.6@sha256:b1fa224cc6c7ff32455e0b1fd9cbfd3d3bc87ecaa8fcb06961ed1afb3db0f9ae
role: worker
image: kindest/node:v1.23.6@sha256:b1fa224cc6c7ff32455e0b1fd9cbfd3d3bc87ecaa8fcb06961ed1afb3db0f9ae
role: worker
image: kindest/node:v1.23.6@sha256:b1fa224cc6c7ff32455e0b1fd9cbfd3d3bc87ecaa8fcb06961ed1afb3db0f9ae

+++++
kubectl apply -f ./contrib/kubernetes/namespace.yaml
helm install packetstreamer ./contrib/helm/ --namespace packetstreamer

+++
root@abasit-PC:~/kind# k get pods -n packetstreamer
NAME READY STATUS RESTARTS AGE
packetstreamer-receiver-596fb5dddf-bp9sc 1/1 Running 0 37m
packetstreamer-sensor-clh4w 1/1 Running 0 37m
packetstreamer-sensor-k89jn 1/1 Running 0 37m

+++++++

root@abasit-PC:~# k logs packetstreamer-sensor-clh4w -n packetstreamer -f
2022/07/18 07:25:43 Start sending
2022/07/18 07:25:43 Now waiting in main
2022/07/18 07:26:43 Total packets read from interface is 27
2022/07/18 07:27:43 Total packets read from interface is 28
2022/07/18 07:28:43 Total packets read from interface is 28
2022/07/18 07:29:43 Total packets read from interface is 28
2022/07/18 07:30:43 Total packets read from interface is 29
2022/07/18 07:31:43 Total packets read from interface is 29
2022/07/18 07:32:43 Total packets read from interface is 29
2022/07/18 07:33:43 Total packets read from interface is 29
2022/07/18 07:34:43 Total packets read from interface is 30
2022/07/18 07:35:43 Total packets read from interface is 30
2022/07/18 07:36:43 Total packets read from interface is 30
2022/07/18 07:37:43 Total packets read from interface is 30
2022/07/18 07:38:43 Total packets read from interface is 30
2022/07/18 07:39:43 Total packets read from interface is 30
2022/07/18 07:40:43 Total packets read from interface is 30
2022/07/18 07:41:43 Total packets read from interface is 30
2022/07/18 07:42:43 Total packets read from interface is 31
2022/07/18 07:43:43 Total packets read from interface is 31

++++++

root@abasit-PC:~/kind/PacketStreamer# k logs packetstreamer-receiver-596fb5dddf-bp9sc -n packetstreamer -f
2022/07/18 07:25:45 Start receiving
2022/07/18 07:25:45 Now waiting in main

2022/07/18 07:26:45 Total data transfer size is 0 B
2022/07/18 07:27:45 Total data transfer size is 0 B
2022/07/18 07:28:45 Total data transfer size is 0 B
2022/07/18 07:29:45 Total data transfer size is 0 B
2022/07/18 07:30:45 Total data transfer size is 0 B
2022/07/18 07:31:45 Total data transfer size is 0 B
2022/07/18 07:32:45 Total data transfer size is 0 B
2022/07/18 07:33:45 Total data transfer size is 0 B
2022/07/18 07:34:45 Total data transfer size is 0 B
2022/07/18 07:35:45 Total data transfer size is 0 B
2022/07/18 07:36:45 Total data transfer size is 0 B
2022/07/18 07:37:45 Total data transfer size is 0 B
2022/07/18 07:38:45 Total data transfer size is 0 B
2022/07/18 07:39:45 Total data transfer size is 0 B
2022/07/18 07:40:45 Total data transfer size is 0 B
2022/07/18 07:41:45 Total data transfer size is 0 B
2022/07/18 07:42:45 Total data transfer size is 0 B
2022/07/18 07:43:45 Total data transfer size is 0 B

++
root@abasit-PC:~# k exec -it packetstreamer-receiver-596fb5dddf-bp9sc -n packetstreamer sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # cd tmp/
/tmp # ls
dump
/tmp # ls -l
total 4
-rw-r--r-- 1 root root 24 Jul 18 07:25 dump
/tmp # ls -l
total 4
-rw-r--r-- 1 root root 24 Jul 18 07:25 dump
/tmp #

Expected behavior
I should be seeing the packet arriving at the receiver and the size of the /tmp/dump should increase

Additional context
I have tried the similar installation on kubeadm Calico cluster but facing the same issue.

confluent-kafka-go doesn't build with musl

 #15 28.20 # github.com/confluentinc/confluent-kafka-go/kafka
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_txnmgr.o): in function `rd_kafka_txn_set_fatal_error':
#15 28.20 (.text+0x152): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_txnmgr.o): in function `rd_kafka_txn_set_abortable_error0':
#15 28.20 (.text+0x692): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_admin.o): in function `rd_kafka_NewTopic_new':
#15 28.20 (.text+0x315): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_admin.o): in function `rd_kafka_AlterConfigsResponse_parse':
#15 28.20 (.text+0x721a): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_admin.o): in function `rd_kafka_admin_result_fail':
#15 28.20 (.text+0x8f6c): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_admin.o):(.text+0x9cfb): more undefined references to `__strdup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_metadata.o): in function `rd_kafka_parse_Metadata':
#15 28.20 (.text+0x559b): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_metadata.o): in function `rd_list_string_copy':
#15 28.20 (.text+0x80d2): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_metadata.o): in function `rd_kafka_metadata_leader_query_tmr_cb':
#15 28.20 (.text+0x8f03): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o): in function `rd_kafka_topic_partition_list_add0':
#15 28.20 (.text+0x311a): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o): in function `rd_kafka_toppar_enq_error':
#15 28.20 (.text+0x53c6): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o): in function `rd_kafka_topic_partition_new':
#15 28.20 (.text+0x55e5): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o):(.text+0x5665): more undefined references to `__strdup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o): in function `rd_kafka_topic_partition_new_from_rktp':
#15 28.20 (.text+0x5778): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o): in function `rd_kafka_topic_partition_list_get_leaders':
#15 28.20 (.text+0x75a3): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_partition.o): in function `rd_kafka_topic_partition_list_get_topic_names':
#15 28.20 (.text+0x79cb): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_pattern.o): in function `rd_kafka_pattern_new':
#15 28.20 (.text+0xa1): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_pattern.o): in function `rd_kafka_pattern_list_init':
#15 28.20 (.text+0x37f): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_pattern.o): in function `rd_kafka_pattern_list_new':
#15 28.20 (.text+0x5c9): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_cgrp.o): in function `rd_kafka_cgrp_op_serve':
#15 28.20 (.text+0xc52): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_cgrp.o): in function `rd_kafka_cgrp_assigned_offsets_commit':
#15 28.20 (.text+0x4e4a): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_cgrp.o): in function `rd_kafka_consumer_group_metadata_new_with_genid':
#15 28.20 (.text+0x7acf): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x7ae3): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x7af9): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_cgrp.o): in function `rd_kafka_consumer_group_metadata_dup':
#15 28.20 (.text+0x7be2): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_cgrp.o):(.text+0x7bfa): more undefined references to `__strdup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_request.o): in function `rd_kafka_handle_OffsetFetch':
#15 28.20 (.text+0x6308): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_request.o): in function `rd_kafka_MetadataRequest':
#15 28.20 (.text+0xd216): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_request.o): in function `rd_list_string_copy':
#15 28.20 (.text+0xdad2): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_op.o): in function `rd_kafka_q_op_err':
#15 28.20 (.text+0x960): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_op.o): in function `rd_kafka_consumer_err':
#15 28.20 (.text+0xab7): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_op.o): in function `rd_kafka_op_throttle_time':
#15 28.20 (.text+0x1361): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_offset.o):(.text+0x209): more undefined references to `__strdup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka.o): in function `rd_kafka_ListGroups_resp_cb':
#15 28.20 (.text+0x7905): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka.o): in function `rd_kafka_DescribeGroups_resp_cb':
#15 28.20 (.text+0xc6f6): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xc74d): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xc781): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xc7b0): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xc7dc): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xcdee): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka.o):(.text+0xce26): more undefined references to `__strndup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_plugin.o): in function `rd_kafka_plugins_conf_set0':
#15 28.20 (.text+0x22e): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rddl.o): in function `rd_dl_error':
#15 28.20 (.text+0x1af): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x1df): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdhdrhistogram.o): in function `unittest_rdhdrhistogram':
#15 28.20 (.text+0x1e92): undefined reference to `__isnan'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x1ea9): undefined reference to `__isnan'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sasl_oauthbearer.o): in function `rd_kafka_oauthbearer_set_token0':
#15 28.20 (.text+0x1e4): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x209): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sasl_oauthbearer.o): in function `rd_kafka_oauthbearer_set_token_failure0':
#15 28.20 (.text+0x4db): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sasl_oauthbearer.o): in function `rd_kafka_oauthbearer_unsecured_token0':
#15 28.20 (.text+0x7a0): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x8d1): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sasl_oauthbearer.o):(.text+0x922): more undefined references to `__strdup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sasl_oauthbearer.o): in function `rd_kafka_sasl_oauthbearer_fsm':
#15 28.20 (.text+0x39ec): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock.o): in function `rd_kafka_mock_topic_new':
#15 28.20 (.text+0xcdb): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock.o): in function `rd_kafka_mock_topic_set_error':
#15 28.20 (.text+0x1be3): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock.o): in function `rd_kafka_mock_topic_create':
#15 28.20 (.text+0x1c77): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock.o): in function `rd_kafka_mock_partition_set_leader':
#15 28.20 (.text+0x1d17): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock.o): in function `rd_kafka_mock_partition_set_follower':
#15 28.20 (.text+0x1da7): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock.o):(.text+0x1e3d): more undefined references to `__strdup' follow
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(async.o): in function `async_fibre_swapcontext':
#15 28.20 (.text+0x9d): undefined reference to `setcontext'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(async_posix.o): in function `ASYNC_is_capable':
#15 28.20 (.text+0xb): undefined reference to `getcontext'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(async_posix.o): in function `async_fibre_makecontext':
#15 28.20 (.text+0x3f): undefined reference to `getcontext'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x91): undefined reference to `makecontext'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock_cgrp.o): in function `rd_kafka_mock_cgrp_member_add':
#15 28.20 (.text+0x9db): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xa17): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock_cgrp.o): in function `rd_kafka_mock_cgrp_get':
#15 28.20 (.text+0xdde): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xe04): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock_cgrp.o): in function `rd_kafka_mcgrp_rebalance_timer_cb':
#15 28.20 (.text+0x150a): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sticky_assignor.o): in function `ConsumerPair_new':
#15 28.20 (.text+0x3356): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x336f): undefined reference to `__strdup'
#15 28.20 collect2: error: ld returned 1 exit status
#15 ...

#12 [linux/arm64 builder 3/6] RUN git clone --branch libpcap-1.10.1 --depth 1 https://github.com/the-tcpdump-group/libpcap.git /libpcap     && cd /libpcap     && ./configure     --disable-shared     --disable-usb     --disable-netmap     --disable-bluetooth     --disable-dbus     --disable-rdma     && make -j $(nproc)     && make install
#12 27.79 checking for ANSI C header files... yes
#12 31.72 checking for sys/types.h... yes
#12 32.60 checking for sys/stat.h... yes
#12 33.39 checking for stdlib.h... yes
#12 34.17 checking for string.h... yes
#12 35.01 checking for memory.h... yes
#12 CANCELED

#15 [linux/amd64 builder 6/6] RUN make build STATIC=1 RELEASE=0
#15 34.81 make: *** [Makefile:18: build] Error 2
#15 ERROR: process "/bin/sh -c make build STATIC=1 RELEASE=${RELEASE}" did not complete successfully: exit code: 2
------
 > [linux/amd64 builder 6/6] RUN make build STATIC=1 RELEASE=0:
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock_cgrp.o): in function `rd_kafka_mock_cgrp_get':
#15 28.20 (.text+0xdde): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0xe04): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_mock_cgrp.o): in function `rd_kafka_mcgrp_rebalance_timer_cb':
#15 28.20 (.text+0x150a): undefined reference to `__strndup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: /root/go/pkg/mod/github.com/confluentinc/[email protected]/kafka/librdkafka_vendor/librdkafka_glibc_linux.a(rdkafka_sticky_assignor.o): in function `ConsumerPair_new':
#15 28.20 (.text+0x3356): undefined reference to `__strdup'
#15 28.20 /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: (.text+0x336f): undefined reference to `__strdup'
#15 28.20 collect2: error: ld returned 1 exit status
#15 34.81 make: *** [Makefile:18: build] Error 2

request a release

I wonder if there is any release plan down the path.

README - Add instructions to show live stream use cases

Show how to live stream traffic into various tools for detection purposes.

I.e. sensor --> receiver --> live stream | TOOL where TOOL = { Zeek, Suricata, Tshark, Moloch etc}

Suricata
Zeek
Tshark
Moloch

Support output to stdout

Output file is defined by the configuration output: file: path: filename

There are use cases where you might want to write to stdout, such as in stream processing where you don't want to maintain a large file on disk:

./packet-streamer receiver --config contrib/receiver-stdout.yaml | tcpdump -r - | grep something

You can set the filename to /dev/stdout, but this has limitations if the user has changed permissions (example).

Feature request: allow a user to write to stdout. For example, we could support a special filename '-' or 'stdout', or a different configuration output: file: stdout

Stream packets to S3 buckets

Essentially
sensor -> receiver -> S3

unable to capture network interface

Describe the bug
A clear and concise description of what the bug is.
linux: centos7

start receiver which listen 8081 port

./packetstreamer receiver --config ./contrib/config/receiver.yaml

start sensor

./packetstreamer sensor --config ./contrib/config/sensor-local.yaml

2022/04/09 19:13:29 Start receiving
2022/04/09 19:13:29 Now waiting in main
2022/04/09 19:13:31 Accepted connection on socket: tcp 127.0.0.1:47994
2022/04/09 19:14:29 Total data transfer size is 0 B
2022/04/09 19:15:29 Total data transfer size is 0 B
2022/04/09 19:16:29 Total data transfer size is 0 B
2022/04/09 19:17:29 Total data transfer size is 0 B

send http request to localhost server listen on 9100, packetstreamer transfer data is 0.

can't get tls mode to work?

Hi there

I just saw this released and had to give it a go. I got it working in non-tls mode (ie sensor pushing it's network data to a remote receiver), but when I added the tls lines to the before-working receiver.yaml it broke

input:
  address: 0.0.0.0
  port: 999
tls:
   enable: true
   certFile: /path/file.pem
   keyFile: /path/file.key
output:
  file:
    path: /var/spool/packstreamer/dump.pcap

It errors out with

2022/04/03 08:46:42 Start receiving
2022/04/03 08:46:42 Now waiting in main
2022/04/03 08:46:42 open : no such file or directory

I ran it via strace and the "open : " is

<... openat resumed>) = -1 ENOENT (No such file or directory)

ie no filename, no file descriptor either? Remove those lines and it works again - sans tls of course. BTW it doesn't exit - but it doesn't listen on the port either - so it's dead.

strace also shows no attempt to open the certFile/keyFile

Any ideas? This is compiled and run under Fedora-35 with golang-1.16.15-1.fc35.x86_64

Thanks

Jason

Stream packets to Redis

Essentially

sensor -> receiver -> Redis

Update diagram to show K8s, Fargate and multi-cloud capture and collection

Need help in changing the diagram to show various types of infra that PacketStreamer supports, the same needs to be added to deployment instructions as well.

Essentially, show all potential sources of capture and all planned sinks.

Stream packets to Kafka

Essentially
sensor -> receiver -> Kafka

invalid memory address or nil pointer dereference

Describe the bug
When i installed packetstreamer to one of our AWS kubernetes clusters using provided helm-chart with receiver.yaml values file (also provided under contrib/config)

I am getting below error on the both receiver and sensor pods

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x62b655]

goroutine 1 [running]:
github.com/deepfence/PacketStreamer/pkg/config.NewConfig({0x7ffea2bcfcd5, 0x1f})
/src/pkg/config/config.go:138 +0x175
github.com/deepfence/PacketStreamer/cmd.initConfig()
/src/cmd/root.go:46 +0x4d
github.com/spf13/cobra.(*Command).preRun(0xe9f800)
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:886 +0x42
github.com/spf13/cobra.(*Command).execute(0xe9f800, {0xc0000d5280, 0x2, 0x2})
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:822 +0x1db
github.com/spf13/cobra.(*Command).ExecuteC(0xe9f580)
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/deepfence/PacketStreamer/cmd.Execute()
/src/cmd/root.go:28 +0x25
main.main()
/src/main.go:6 +0x17

I didn't change default values file and using this image deepfenceio/deepfence_packetstreamer:latest

To Reproduce
Steps to reproduce the behavior:
helm install to an kubernetes cluster in aws using receiver.yaml values file

Expected behavior
was expecting it to work out of box

Additional context
Add any other context about the problem here.

Explore alternatives to TLV, handle cases where compressed pcap block > original pcap block

Current handcoded TLV works as a simple and lightweight implementation for this specific use case, for the sake of completeness we should explore alternatives (protobuf etc) and document pros and cons.

One of the reasons we should do this is ensure things work and are logically consistent even when compressed pcap block is larger than original pcap block (possible due high entropy in pcaps, binary data etc). Current fix introduces another field to indicate whether we are planning to send original pcap or compressed, while the fix works we might be doing other such transformations on pcap block which create larger dest pcap blocks so ability to handle sizes > original pcap size is required. Compression is just one of the actions on pcap blocks, there could be more in future.

Custom BPF filters on PacketStreamer Sensor

Is your feature request related to a problem? Please describe.
Currently, BPF filters are set based on ports (passed as configuration) inside PacketStreamer sensor and doesn't allow any other custom BPF filter to be used.

Describe the solution you'd like
custom BPF filters should be taken as config input for sensor.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.