possible missing event fields about py-idstools HOT 10 CLOSED

I'm trying to replicate, kind of, the info that Barnyard2 provides in database tables, but
certain event fields seem to be missing, or I'm not sure how to get them, in idstools.

I'm using:

u2json_reader = unified2.SpoolEventReader(
directory=args.directory,
prefix=args.prefix,
follow=args.follow,
bookmark=args.bookmark)
In the generator, I print out the event dict as received by:

for event in u2json_reader:
. . .
... here's a list of event fields I can't find in the output, and
I've referenced the mysql table/column names as used by barnyard2:

signature:
sig_priority ... is: priority
sig_class_id ... is: signature-id ?
sig_class_name
sig_rev ... is: signature-revision
sig_sid
sig_gid

sig_priority -> priority
sig_class_id -> classification-id
sig_class_name -> not provided
sig_rev -> signature-revision
sig_sid -> signature-id
sig_gid -> generator-id

A unified2 file does not include the sig_class_name, Barnyard2 uses the classification id to lookup the classification name int he classification file (which is why you tell Barnyard2 where to find this file on startup).

idstools can help a little, see http://idstools.readthedocs.org/en/latest/maps.html#classificationmap which will help looking the classification.conf into a lookup table. The u2json tools uses this, and I also discuss it a little here http://blog.jasonish.org/category/unified2 .

iphdr:
ip_ver
ip_hlen
ip_tos
ip_len
ip_id
ip_flags
ip_off
ip_ttl
ip_proto ... is: proto ?
ip_csum

tcphdr:
tcp_seq
tcp_ack
tcp_off
tcp_res
tcp_flags
tcp_win
tcp_csum
tcp_urp

udphdr: (not sure about these)
udp_len
udp_csum

icmphdr:
icmp_type ... is seen in output
icmp_code ... is seen in output
icmp_csum
icmp_id
icmp_seq
Suggestions?

These extra UDP, TCP and ICMP fields aren't broken out in a unified2 log file, instead you have to extract them from the packet data. With idstools that means getting the packet (event["packets"][0].data) and decoding the header fields to get at the specific pieces of information. Idstools does not have code to decode the packet into fields so you are left to figure that out for yourself. However, after looking around, I should probably add some lightweight code to do that. You could use Scapy though:

from idstools import unified2
import socket
from scapy.all import *

reader = unified2.FileEventReader("./tests/merged.log")
for event in reader:
    data = event["packets"][0]["data"]
    ether = Ether(data)
    ip = ether/IP()
    print(ip.version)
    print(ip.ihl)
    print(ip.tos)
    print(ip.len)
    print(ip.id)
    print(ip.flags)
    print(ip.ttl)
    print(ip.proto)
    print(ip.chksum)

    if ip.proto == socket.IPPROTO_TCP:
        tcp = ip/TCP()
        print(tcp.seq)
        print(tcp.ack)

    break

from py-idstools.

I overlooked the class name which I am getting via u2json. I will try out scapy. I really don't think
the analysts use those extra UDP, TCP, and ICMP fields, but as I assume that they will probably
notice them missing. Thanks again as idstools is working really well together with elasticsearch-py to bulk index events.

from py-idstools.

I've rarely found a need to index them into a database, especially if the packet is there - as you can then break them out for visualization purposes in the reporting tool.

Anyways, I'm scratching an itch making a lightweight decoder. Time depending, I may have something today.

from py-idstools.

Thanks ... this is all coming together much better than I expected ... it's nice when it just works.

from py-idstools.

If interested, check out https://github.com/jasonish/py-idstools/blob/master/idstools/packet.py (which I've added to master). It handles ethernet packets, and will decode IPv4, TCP, UDP and ICMP. Simple example would be:

import sys

from idstools import unified2
from idstools import packet

reader = unified2.FileEventReader(sys.argv[1])
for event in reader:
    if not event["packets"]:
        continue
    data = event["packets"][0]["data"]
    pkt = packet.decode_ethernet(data)
    print(pkt)
    break

So far its just an excercise in what the API should look like, as well as what the object returned should look like.

from py-idstools.

I noticed that any reference to event["packets"] is always [0] is that because there is never more than one ? The new module packet.py is exactly what I needed ... thanks.

from py-idstools.

There can often be multiple packets logged per event, so depending on the database layout, you may just want to get the parameters off the first one, or dig deeper. My EveBox will display multiple packets for an event if there is more than one.

Also keep in mind that I've seen unified2 events with no packet - so handle that case as well.

from py-idstools.

Also, I noticed that newer versions of Snort are capturing IPv6 addresses, so I'm wondering if those
packets have any of the these fields. For now, I just store them in elasticsearch as a string.
Also, I think I saw someone that Barnyard2 was being upgraded to handle IPv6 and have a new database schema. Just curious about all of this.

from py-idstools.

IPv6 will have similar fields, but I haven't added it to packets.py. I will soon - the IPv4 code was pretty much cut and paste from another project, but I don't have existing similar code for IPv6.

from py-idstools.

I will keep an eye out for that, thanks ... or don't hesitate to let me know as I don't mind trying to use it.

from py-idstools.