Comments (7)
jasonish
commented on June 21, 2024
This does not include a tool for converting unified2 to pcap. There is a tool to convert Suricata eve.json to pcap though.
from py-idstools.
DDB-en
commented on June 21, 2024
Yes i know but i think there is some issues in 'data' (i mean for example packets[0]['data'])
Because i build a pcap file (base on idstools-eve2pcap method) and i found some errors in packets (or maybe my fault)
This is my code (summary):
class Pcap:
#same as idstools-eve2pcap
def eve2pcap(event):
if not "data" in event:
return None, None
packet = event["data"]
hdr = pcap_pkthdr()
hdr.ts_sec, hdr.ts_usec = event["packet-second"], event["packet-microsecond"]
hdr.pktlen = len(packet)
hdr.caplen = len(packet)
return (hdr, packet)
linktype=event[0]['linktype']
pcap = Pcap.open_dead(linktype, 65535)
dumper = pcap.dump_open('/tmp/1.pcap')
for p in event:
hdr, packet = eve2pcap(p)
if hdr and packet:
dumper.dump(hdr, ctypes.c_char_p(packet))
dumper.close()
Unified decoded record:
{'extra-data': [], 'mpls-label': None, 'destination-ip': '104.66.89.155', 'signature-revision': 2, 'vlan-id': None, 'signature-id': 2027390, 'protocol': 6, 'classification-id': 2, 'sport-itype': 50990, 'priority': 3, 'appid': None, 'dport-icode': 80, 'blocked': 0, 'impact': 0, 'generator-id': 1, 'source-ip': '192.168.70.25', 'impact-flag': 0, 'source-ip.raw': b'\xc0\xa8F\x19', 'destination-ip.raw': b'hBY\x9b', 'sensor-id': 0, 'event-id': 280, 'packets': [{'sensor-id': 0, 'length': 400, 'data': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00E\x00\x01\x82\x00\x00\x00\x00\x00\x06\xf0\xd7\xc0\xa8F\x19hBY\x9b\xc7.\x00P\x00\x00\x00\x00\x00\x00\x00\x00P\x00\x00\x00\xb2q\x00\x00POST /fwlink/?LinkID=252669&clcid=0x409 HTTP/1.1\r\nConnection: Keep-Alive\r\nContent-Type: text/xml; charset="UTF-16LE"\r\nUser-Agent: MICROSOFT_DEVICE_METADATA_RETRIEVAL_CLIENT\r\nSOAPAction: "http://schemas.microsoft.com/windowsmetadata/services/2007/09/18/dms/DeviceMetadataService/GetDeviceMetadata"\r\nContent-Length: 1420\r\nHost: go.microsoft.com\r\n\r\n', 'packet-microsecond': 742421, 'linktype': 1, 'event-id': 280, 'event-second': 1602229464, 'packet-second': 1602229464}, {'sensor-id': 0, 'length': 1474, 'data': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00E\x00\x05\xb4\x00\x00\x00\x00\x00\x06\xec\xa5\xc0\xa8F\x19hBY\x9b\xc7.\x00P\x00\x00\x00\x00\x00\x00\x00\x00P\x00\x00\x00\xc6K\x00\x00\xff\xfe<\x00?\x00x\x00m\x00l\x00 \x00v\x00e\x00r\x00s\x00i\x00o\x00n\x00=\x00"\x001\x00.\x000\x00"\x00 \x00e\x00n\x00c\x00o\x00d\x00i\x00n\x00g\x00=\x00"\x00U\x00T\x00F\x00-\x001\x006\x00"\x00?\x00>\x00<\x00s\x00:\x00E\x00n\x00v\x00e\x00l\x00o\x00p\x00e\x00 \x00x\x00m\x00l\x00n\x00s\x00:\x00s\x00=\x00"\x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00s\x00c\x00h\x00e\x00m\x00a\x00s\x00.\x00x\x00m\x00l\x00s\x00o\x00a\x00p\x00.\x00o\x00r\x00g\x00/\x00s\x00o\x00a\x00p\x00/\x00e\x00n\x00v\x00e\x00l\x00o\x00p\x00e\x00/\x00"\x00>\x00<\x00s\x00:\x00H\x00e\x00a\x00d\x00e\x00r\x00>\x00<\x00h\x00:\x00c\x00d\x00 \x00x\x00m\x00l\x00n\x00s\x00:\x00h\x00=\x00"\x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00s\x00c\x00h\x00e\x00m\x00a\x00s\x00.\x00m\x00i\x00c\x00r\x00o\x00s\x00o\x00f\x00t\x00.\x00c\x00o\x00m\x00/\x00w\x00i\x00n\x00d\x00o\x00w\x00s\x00m\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00/\x00s\x00e\x00r\x00v\x00i\x00c\x00e\x00s\x00/\x002\x000\x000\x007\x00/\x000\x009\x00/\x001\x008\x00/\x00d\x00m\x00s\x00"\x00>\x00<\x00h\x00:\x00c\x00v\x00>\x001\x000\x00.\x000\x00.\x001\x009\x000\x004\x001\x00<\x00/\x00h\x00:\x00c\x00v\x00>\x00<\x00h\x00:\x00c\x00c\x00>\x00U\x00S\x00A\x00<\x00/\x00h\x00:\x00c\x00c\x00>\x00<\x00/\x00h\x00:\x00c\x00d\x00>\x00<\x00/\x00s\x00:\x00H\x00e\x00a\x00d\x00e\x00r\x00>\x00<\x00s\x00:\x00B\x00o\x00d\x00y\x00>\x00<\x00D\x00e\x00v\x00i\x00c\x00e\x00M\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00B\x00a\x00t\x00c\x00h\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00 \x00x\x00m\x00l\x00n\x00s\x00=\x00"\x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00s\x00c\x00h\x00e\x00m\x00a\x00s\x00.\x00m\x00i\x00c\x00r\x00o\x00s\x00o\x00f\x00t\x00.\x00c\x00o\x00m\x00/\x00w\x00i\x00n\x00d\x00o\x00w\x00s\x00m\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00/\x00s\x00e\x00r\x00v\x00i\x00c\x00e\x00s\x00/\x002\x000\x000\x007\x00/\x000\x009\x00/\x001\x008\x00/\x00d\x00m\x00s\x00"\x00>\x00<\x00L\x00o\x00c\x00L\x00i\x00s\x00t\x00>\x00<\x00l\x00o\x00c\x00>\x00M\x00u\x00l\x00t\x00i\x00L\x00o\x00c\x00<\x00/\x00l\x00o\x00c\x00>\x00<\x00l\x00o\x00c\x00>\x00e\x00n\x00-\x00U\x00S\x00<\x00/\x00l\x00o\x00c\x00>\x00<\x00l\x00o\x00c\x00>\x00e\x00n\x00<\x00/\x00l\x00o\x00c\x00>\x00<\x00/\x00L\x00o\x00c\x00L\x00i\x00s\x00t\x00>\x00<\x00M\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00g\x00d\x00m\x00d\x00m\x00i\x00d\x00>\x00<\x00r\x00i\x00d\x00>\x001\x00<\x00/\x00r\x00i\x00d\x00>\x00<\x00m\x00i\x00d\x00>\x003\x00E\x005\x00B\x005\x00E\x00A\x009\x00-\x005\x007\x005\x005\x00-\x005\x00B\x00C\x007\x00-\x00B\x001\x00B\x00C\x00-\x00F\x00A\x003\x006\x007\x001\x001\x00B\x006\x00C\x002\x008\x00<\x00/\x00m\x00i\x00d\x00>\x00<\x00/\x00g\x00d\x00m\x00d\x00m\x00i\x00d\x00>\x00<\x00/\x00M\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00H\x00W\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00g\x00d\x00m\x00d\x00h\x00w\x00i\x00d\x00>\x00<\x00r\x00i\x00d\x00>\x001\x00<\x00/\x00r\x00i\x00d\x00>\x00<\x00h\x00w\x00i\x00d\x00s\x00>\x00<\x00h\x00w\x00i\x00d\x00>\x00D\x00O\x00I\x00D\x00:\x00M\x00O\x00N\x00I\x00T\x00O\x00R\x00\\\x00D\x00e\x00f\x00a\x00u\x00l\x00t\x00_\x00M\x00o\x00n\x00i\x00t\x00o\x00r\x00<\x00/\x00h\x00w\x00i\x00d\x00>\x00<\x00/\x00h\x00w\x00i\x00d\x00s\x00>\x00<\x00/\x00g\x00d\x00m\x00d\x00h\x00w\x00i\x00d\x00>\x00<\x00/\x00H\x00W\x00I\x00D\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00>\x00<\x00/\x00D\x00e\x00v\x00i\x00c\x00e\x00M\x00e\x00t\x00a\x00d\x00a\x00t\x00a\x00B\x00a\x00t\x00c\x00h\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00>\x00<\x00/\x00s\x00:\x00B\x00o\x00d\x00y\x00>\x00<\x00/\x00s\x00:\x00E\x00n\x00v\x00e\x00l\x00o\x00p\x00e\x00>\x00', 'packet-microsecond': 742421, 'linktype': 1, 'event-id': 280, 'event-second': 1602229464, 'packet-second': 1602229464}], 'event-microsecond': 742421, 'event-second': 1602229464}
This is the output file:
265750498.zip
also this record is same as above record in eve.json:
{"timestamp":"2020-10-09T11:14:24.742421","flow_id":12984035864390,"in_iface":"eth0","event_type":"alert","src_ip":"192.168.70.25","src_port":50990,"dest_ip":"104.66.89.155","dest_port":80,"proto":"TCP","metadata":{"flowbits":["FB180732_0"]},"tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2027390,"rev":2,"signature":"ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent","category":"Unknown Traffic","severity":3,"metadata":{"updated_at":["2019_09_28"],"signature_severity":["Minor"],"performance_impact":["Low"],"former_category":["USER_AGENTS"],"deployment":["Perimeter"],"created_at":["2019_05_28"],"attack_target":["Client_Endpoint"],"affected_product":["Web_Browsers"]}},"http":{"hostname":"go.microsoft.com","url":"\/fwlink\/?LinkID=252669&clcid=0x409","http_user_agent":"MICROSOFT_DEVICE_METADATA_RETRIEVAL_CLIENT","http_method":"POST","protocol":"HTTP\/1.1","length":0},"app_proto":"http","flow":{"pkts_toserver":9,"pkts_toclient":8,"bytes_toserver":4060,"bytes_toclient":1072,"start":"2020-10-09T11:14:24.618310"},"payload":"UE9TVCAvZndsaW5rLz9MaW5rSUQ9MjUyNjY5JmNsY2lkPTB4NDA5IEhUVFAvMS4xDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQpDb250ZW50LVR5cGU6IHRleHQveG1sOyBjaGFyc2V0PSJVVEYtMTZMRSINClVzZXItQWdlbnQ6IE1JQ1JPU09GVF9ERVZJQ0VfTUVUQURBVEFfUkVUUklFVkFMX0NMSUVOVA0KU09BUEFjdGlvbjogImh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vd2luZG93c21ldGFkYXRhL3NlcnZpY2VzLzIwMDcvMDkvMTgvZG1zL0RldmljZU1ldGFkYXRhU2VydmljZS9HZXREZXZpY2VNZXRhZGF0YSINCkNvbnRlbnQtTGVuZ3RoOiAxNDIwDQpIb3N0OiBnby5taWNyb3NvZnQuY29tDQoNCv\/+PAA\/AHgAbQBsACAAdgBlAHIAcwBpAG8AbgA9ACIAMQAuADAAIgAgAGUAbgBjAG8AZABpAG4AZwA9ACIAVQBUAEYALQAxADYAIgA\/AD4APABzADoARQBuAHYAZQBsAG8AcABlACAAeABtAGwAbgBzADoAcwA9ACIAaAB0AHQAcAA6AC8ALwBzAGMAaABlAG0AYQBzAC4AeABtAGwAcwBvAGEAcAAuAG8AcgBnAC8AcwBvAGEAcAAvAGUAbgB2AGUAbABvAHAAZQAvACIAPgA8AHMAOgBIAGUAYQBkAGUAcgA+ADwAaAA6AGMAZAAgAHgAbQBsAG4AcwA6AGgAPQAiAGgAdAB0AHAAOgAvAC8AcwBjAGgAZQBtAGEAcwAuAG0AaQBjAHIAbwBzAG8AZgB0AC4AYwBvAG0ALwB3AGkAbgBkAG8AdwBzAG0AZQB0AGEAZABhAHQAYQAvAHMAZQByAHYAaQBjAGUAcwAvADIAMAAwADcALwAwADkALwAxADgALwBkAG0AcwAiAD4APABoADoAYwB2AD4AMQAwAC4AMAAuADEAOQAwADQAMQA8AC8AaAA6AGMAdgA+ADwAaAA6AGMAYwA+AFUAUwBBADwALwBoADoAYwBjAD4APAAvAGgAOgBjAGQAPgA8AC8AcwA6AEgAZQBhAGQAZQByAD4APABzADoAQgBvAGQAeQA+ADwARABlAHYAaQBjAGUATQBlAHQAYQBkAGEAdABhAEIAYQB0AGMAaABSAGUAcQB1AGUAcwB0ACAAeABtAGwAbgBzAD0AIgBoAHQAdABwADoALwAvAHMAYwBoAGUAbQBhAHMALgBtAGkAYwByAG8AcwBvAGYAdAAuAGMAbwBtAC8AdwBpAG4AZABvAHcAcwBtAGUAdABhAGQAYQB0AGEALwBzAGUAcgB2AGkAYwBlAHMALwAyADAAMAA3AC8AMAA5AC8AMQA4AC8AZABtAHMAIgA+ADwATABvAGMATABpAHMAdAA+ADwAbABvAGMAPgBNAHUAbAB0AGkATABvAGMAPAAvAGwAbwBjAD4APABsAG8AYwA+AGUAbgAtAFUAUwA8AC8AbABvAGMAPgA8AGwAbwBjAD4AZQBuADwALwBsAG8AYwA+ADwALwBMAG8AYwBMAGkAcwB0AD4APABNAEkARABSAGUAcQB1AGUAcwB0AHMAPgA8AGcAZABtAGQAbQBpAGQAPgA8AHIAaQBkAD4AMQA8AC8AcgBpAGQAPgA8AG0AaQBkAD4AMwBFADUAQgA1AEUAQQA5AC0ANQA3ADUANQAtADUAQgBDADcALQBCADEAQgBDAC0ARgBBADMANgA3ADEAMQBCADYAQwAyADgAPAAvAG0AaQBkAD4APAAvAGcAZABtAGQAbQBpAGQAPgA8AC8ATQBJAEQAUgBlAHEAdQBlAHMAdABzAD4APABIAFcASQBEAFIAZQBxAHUAZQBzAHQAcwA+ADwAZwBkAG0AZABoAHcAaQBkAD4APAByAGkAZAA+ADEAPAAvAHIAaQBkAD4APABoAHcAaQBkAHMAPgA8AGgAdwBpAGQAPgBEAE8ASQBEADoATQBPAE4ASQBUAE8AUgBcAEQAZQBmAGEAdQBsAHQAXwBNAG8AbgBpAHQAbwByADwALwBoAHcAaQBkAD4APAAvAGgAdwBpAGQAcwA+ADwALwBnAGQAbQBkAGgAdwBpAGQAPgA8AC8ASABXAEkARABSAGUAcQB1AGUAcwB0AHMAPgA8AC8ARABlAHYAaQBjAGUATQBlAHQAYQBkAGEAdABhAEIAYQB0AGMAaABSAGUAcQB1AGUAcwB0AD4APAAvAHMAOgBCAG8AZAB5AD4APAAvAHMAOgBFAG4AdgBlAGwAbwBwAGUAPgA=","stream":1,"packet":"AFBWhcYcxBL1MguzCABFAAAouHVAAIAGebvAqEYZaEJZm8cuAFAHVxwRqyIFNlARAgBJ9QAAAAAAAAAA","packet_info":{"linktype":1}}
This pcap file is generated by idstools-eve2pcap:
1.zip
as you can see in first file there is some issues.
please help me to fixed them,
Thanks
from py-idstools.
jasonish
commented on June 21, 2024
Can you describe what is wrong, and what you expect?
On brief look, in 265750498.zip I see an extra packet. I'm not sure where that is coming from. It looks like 1.zip is a pcap generated from eve using the --payload option? Is that correct?
One difference to note is that the payload in an eve record contains re-assembled data, perhaps from multiple TCP packets and contains no packet headers, so the packet headers are reconstructed by the eve2pcap tool.
Unified2 records have the alerting packet, so this is much likely to be smaller than the payload you'd see in the Eve record. The packet data in unified2 also contains the network header, so they're shouldn't be a need to reconstruct it, but there is also no verification that it is correct.
If you're goal is to convert unified2 to pcap, aren't there existing tools out there to do that?
from py-idstools.
DDB-en
commented on June 21, 2024
I expect a pcap file same as eve2pcap output (plus network header),
Yes, it was generated from eve using the --payload option.
And about the wrong things, I have some additional warning and information in wireshark on my pcap rather than eve2pcap output. For instance:
1-
Is it normal?
2-
Some times network header is zero, Is there any wrong on my code?
Both issues occurs on non ssl TCP packets usually.
SSL (TCP) and UDP packets are fine
from py-idstools.
jasonish
commented on June 21, 2024
Code looked fine. Remember that the pcap generation from the Eve payload (Suricata only) crafts a TCP/UDP/IP header which ensures its more or less correct - as the payload field in eve lacks headers.
When using the packet from unified2, the header is taken as-is.
You should look at u2boat and see what its pcap output looks like for comparison.
from py-idstools.
jasonish
commented on June 21, 2024
One other item, I haven't verified if the Unified2 reader is correct anymore. Snort has been known to update it in incompatible ways, or change the output depending on compile time options.
from py-idstools.
DDB-en
commented on June 21, 2024
Thanks for the reply @jasonish.
from py-idstools.
Related Issues (20)
- appStats u2 can't work HOT 1
- Bug: Multiple instances of rule options fields clobber eachother HOT 4
- Recent versions of Snort unified2 not supported. HOT 5
- Feature Request: ability to parse the source, destination, protocol using dictionary.
- SoolRecordReader stop working HOT 2
- eve2pcap.py fails with IPv6 addresses HOT 1
- python2-scapy as pkg dependency
- Connection with suricata-update HOT 2
- Feature request: mutate metadata key value pairs
- Add .md5 extension between URL's filename and its parameters HOT 1
- Provide option for idstools-u2eve to reload sid-msg.map after updating sid-msg.map contents. HOT 1
- Unified2 Event Types mpls, vlan, and appid not included in u2eve output
- u2json event.appid output is in byte format and mangled
- Tests fail with python 3.11 HOT 1
- Rule parsing fails if last option doesn't close with semi-colon HOT 1
- memory usage increase issue HOT 3
- New release to support python 3.13 HOT 6
- Wrong parsing of pcre and possibly others
- Invalid issue
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py-idstools.