Comments (8)
jgc234
commented on June 19, 2024
1
@mehrdadrad - tested and works fine for the last few hours using an SRX with netflow v9 IPv4 only.. removed netflowv9.tempaltes cache file to force correct parsing of transmitted template definitions.
from vflow.
jgc234
commented on June 19, 2024
I think I've worked this one out.. The detector for padding between flowsets broken in vflow/netflow/v9/decoder.go:decodeSet(). It failed to detect the padding, got out-of-sync and started parsing the padding as the next record. .. Once it started eating rubbish, the promotion to an unsigned was wrapping and trying to parse 65K of data (uint16(-1)) .. Currently:
// the next set should be greater than 4 bytes otherwise that's padding
for err == nil && setHeader.Length > uint16(d.reader.ReadCount()-startCount) && d.reader.Len() > 4 {
The "d.reader.Len() > 4" would detect the total bytes remaining in the packet, however we also need to test for bytes remaining in the current flowset (against the header length vs the current offset).
// the next set should be greater than 4 bytes otherwise that's padding
for err == nil && (int(setHeader.Length) - (d.reader.ReadCount()-startCount) > 4) && d.reader.Len() > 4 {
Someone will need to check that logic - I don't know Go or Netflow well.
from vflow.
jgc234
commented on June 19, 2024
Actuallly.. there is a bit of code further down that detects the padding, but it does so by entering the loop again and peeking at the data - if there's zeros there, it bails out.. (rather than checking the current offset position in the FlowSet).. and it only does this for FlowSetID of 0 or 1 (ie templates, not data).
In the packet above, FlowSetID 256 (data) has padding, so it won't get caught by this test..
The IPFIX code uses the same algorithm to peak past the end and hope for zeros.
Is there some magic reason why it's done this way?
from vflow.
jgc234
commented on June 19, 2024
diff --git a/netflow/v9/decoder.go b/netflow/v9/decoder.go
index 8b17136..a9de9e2 100644
--- a/netflow/v9/decoder.go
+++ b/netflow/v9/decoder.go
@@ -435,16 +435,10 @@ func (d *Decoder) decodeSet(mem MemCache, msg *Message) error {
}
// the next set should be greater than 4 bytes otherwise that's padding
- for err == nil && setHeader.Length > uint16(d.reader.ReadCount()-startCount) && d.reader.Len() > 4 {
+ for err == nil && (int(setHeader.Length) - (d.reader.ReadCount()-startCount) > 4) && d.reader.Len() > 4 {
if setId := setHeader.FlowSetID; setId == 0 || setId == 1 {
// Template record or template option record
- // Check if only padding is left in this set. A template id of zero indicates padding bytes, which MUST be zero.
- templateId, err := d.reader.PeekUint16()
- if err == nil && templateId == 0 {
- break
- }
-
tr := TemplateRecord{}
if setId == 0 {
err = tr.unmarshal(d.reader)
from vflow.
mehrdadrad
commented on June 19, 2024
@jgc234 thank you for looking at this issue. I tried netflow during development by nprobe as I don't have any netflow device. I can work on this issue next week.
from vflow.
jgc234
commented on June 19, 2024
@mehrdadrad happy to help.. you could spin up a router in a VM (Juniper vSRX or Cisco CSR100v). or replay some captured packets from someone else. I haven't read the IPFIX to see how it'd different from V9.. I'm unsure if the same padding problem could occur there to.
from vflow.
mehrdadrad
commented on June 19, 2024
@jgc234 can you send me a pcap including templates and samples to my email address: [email protected] ?
from vflow.
mehrdadrad
commented on June 19, 2024
@jgc234 pls try and see how it works w/ your SRX. it changed same as you mentioned.
from vflow.
Related Issues (20)
- Sflow paring issue HOT 3
- CVE-2019-17543 - LZ4 vulnerability
- multinode cluster how to guide missing HOT 1
- Errors in trying to use this project as a library HOT 6
- Vflow support of Kafka Partition key
- memcache.go has possible hash collisions, leading to wrong values saved/retrieved from cache
- vflow_ipfix_udp_packets doesn't increase on stress traffic
- Abandoned project? HOT 16
- sflow packets with padding after sample is not parsed correctly
- sflow packets with sample packets of non-UDP/TCP/ICMP are dropped HOT 1
- nf9/ipfix fields of type String are copied to the JSON output as-is, without handling special charachters HOT 1
- Parsing of sflow SourceID from sample record is wrong
- Proposal - support of extended sflow format HOT 3
- ipfix/nf9 unknown elements cause whole data to be dropped HOT 1
- kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
- UDP InErrors observed in sflow listener under load tesing HOT 1
- [kafka.segmentio] not creating topics
- [kafka] timestamp for all messages is always 01/01/1970, 06:59:59
- installation memo HOT 2
- Vflow does'nt recieve Netflow packets HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vflow.