Comments (4)
puzzled too
from vflow.
In interpret.go, the octetDeltaCount is 4 bytes which is less than t.minLen(), so set this filed with []uint8.
after changing the unsigned64 to unsigned32, the filed do not show up as hex strings.
logstash-netflow plugin use two template config to distinguish netflow V9 and ipfix, maybe vflow should use two model.go too?
// Interpret read data fields based on the type - big endian
func Interpret(b *[]byte, t FieldType) interface{} {
if len(*b) < t.minLen() {
return *b
}
from vflow.
Hello folks,
We are trying to determine what fields in Netflow v9 are reported in Hex. Do you know of a resource on the internet that gives the exact field types for Netflow v9?
If this is an issue due to how vflow is processing the incoming data, do you know if it has been fixed yet?
Thanks
from vflow.
I was also very puzzled by this.
After some digging, I can say this is not a v9 vs ipfix issue.
The fields in nf9 and ipfix define a standard length, like unsigned64, unsigned8, etc. See https://www.iana.org/assignments/ipfix/ipfix.xhtml
In vflow the code has a copy of that table, in rfc5102_model.go.
However, it seems that many implementations out there, and I don't think this is a v9 vs. ipfix thing, but more of a per-vendor decision, they decide to send some fields with sizes smaller than the size declared for it in the RFC.
I'm not sure if the RFC allows this or not, but there seems to be quite a few implementations that do it.
As @lwsbox wrote, there's a check in ipfix/decoder.go and netflow/v9/decoder.go which reads bytes from the packet based on the length that appeared in the template, but tries to interpret them as types defined in the RFC. If the length in the packet is smaller than the length specified in the RFC, then vflow will hold that field as a byte array, and that would serialized in the JSON as a hex string.
So yes, @eksantrik , this is a bug or at least a very annoying behaviour in vflow. If we compare this to Wireshark parser, it doesn't exhibit this odd behaviour.
The fix for this could be a bit tricky for the general case.
For unsigned8/16/32/64 types, the Interpret method could be modified to see if the size is smaller than expected, and if so create a new byte array with zero padding in the beginning, and put in the bytes read from the packet at the 'LSB' part of the new byte array. Need to be careful here with endianity.
For signed8/16/32/64 - not sure what to do regarding the most significant bit which indicates sign, but those seem to be rare.
For special types like IPs, MACs, etc - I've yet to see any pcap with this behaviour for them.
I'll perhaps invest some time to create a fix for this, as I agree this is super-annoying, and causes me grief and more work on my code that reads vflow's output.
WDYT?
from vflow.
Related Issues (20)
- Errors in trying to use this project as a library HOT 6
- Vflow support of Kafka Partition key
- memcache.go has possible hash collisions, leading to wrong values saved/retrieved from cache
- vflow_ipfix_udp_packets doesn't increase on stress traffic
- Abandoned project? HOT 16
- sflow packets with padding after sample is not parsed correctly
- sflow packets with sample packets of non-UDP/TCP/ICMP are dropped HOT 1
- nf9/ipfix fields of type String are copied to the JSON output as-is, without handling special charachters HOT 1
- Parsing of sflow SourceID from sample record is wrong
- Proposal - support of extended sflow format HOT 3
- ipfix/nf9 unknown elements cause whole data to be dropped HOT 1
- kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
- UDP InErrors observed in sflow listener under load tesing HOT 1
- [kafka.segmentio] not creating topics
- [kafka] timestamp for all messages is always 01/01/1970, 06:59:59
- installation memo HOT 2
- Vflow does'nt recieve Netflow packets HOT 1
- Can't connecting to nsqd HOT 1
- realtime consume data from kafka to clickhouse
- update go.mod
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vflow.