From the docs:
http://idstools.readthedocs.org/en/latest/maps.html
and
http://blog.jasonish.org/2013/07/easy-unified2-file-reading-in-python.html
... figured out that MsgMap should be SignatureMap, but running this:

from idstools import maps
sigmap = maps.SignatureMap()
sigmap.load_generator_map(open("/etc/snort/gen-msg.map"))
sigmap.load_signature_map(open("/etc/snort/community-sid-msg.map"))
print(sigmap.get(1, 2495))
print(sigmap.get(1, 1))

gives the following error:

Traceback (most recent call last):
  File "resolve_event_msg_and_class_name.py", line 6, in <module>
    sigmap.load_signature_map(open("/etc/snort/community-sid-msg.map"))
  File "/usr/local/lib/python2.7/dist-packages/idstools/maps.py", line 136, in load_signature_map
    "sid": int(parts[0]),
ValueError: invalid literal for int() with base 10: ''

I tried lots of gids/sids but no luck ... suggestions ?
Otherwise, I like these tools as it's hard to find snort related stuff ... thanks for this :)

ip-source => source-ip
ip-destination => destination-ip

apparently it changed a while ago.

Right now, the default log level is info, adding -v is debug. A better option for -v is somewhere between debug and info, then perhaps -vv for debug.

One idea:

• Default: Notice logging. Most logs you see now without -v would be notice.
• Info with -v, this would be quite verbose but not low level debugging messages.
• Debug with -vv.

Some rule in snort 2.9.7.2-1 is creating a breaking case in the python module:

INFO: Loaded 25034 rule message map entries.
INFO: Loaded 38 classifications.
Traceback (most recent call last):
File "py-idstools/bin/idstools-u2json", line 12, in
sys.exit(main())
File "/root/py-idstools/idstools/scripts/u2json.py", line 308, in main
as_json = json.dumps(formatter.format(record))
File "/usr/lib64/python2.7/json/init.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xec in position 633: invalid continuation byte

Using the latest bits. If I clear out the unified log and restart it it will work until this sequence is hit again.

I am merging Suricata ET rules on a server where Suricata daemon is not present, but I know which version of it runs on the boxes. However, rulecat doesn't use '-enhanced' ruleset in such a case, it goes with '-1.3' only.

Would it be possible to add an option with Suricata version override? Or just an option instructing rulecat to download and use the enhanced ruleset?

Not an issue really, but I noticed there's something called ruleman ... are there any docs or demos of how to use this feature ? We are considering writing something as most of the available tools are too costly or unfinished open source projects.

Topic :) Thank you.

This is a request encouraged by Fedora packaging policy

https://bugzilla.redhat.com/show_bug.cgi?id=1398369
https://fedoraproject.org/wiki/Packaging:LicensingGuidelines?rd=Packaging/LicensingGuidelines#License_Clarification

An odd one. I don't get this with the one in tests dir:

(Event)
u2spewfoo.py(116):     for row in rows:
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118):         print("\t" + "\t".join(parts))
	sensor id: 0	event id: 1	event second: 1490338878	event microsecond: 955846
u2spewfoo.py(116):     for row in rows:
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118):         print("\t" + "\t".join(parts))
	sig id: 2402000	gen id: 1	revision: 4391	classification: 30
u2spewfoo.py(116):     for row in rows:
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
 --- modulename: u2spewfoo, funcname: print_address
u2spewfoo.py(83):     if len(addr) <= 15:
u2spewfoo.py(84):         return addr
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
 --- modulename: u2spewfoo, funcname: print_address
u2spewfoo.py(83):     if len(addr) <= 15:
u2spewfoo.py(84):         return addr
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118):         print("\t" + "\t".join(parts))
	priority: 2	ip source: 91.223.133.13	ip destination: x.x.x.x
u2spewfoo.py(116):     for row in rows:
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(118):         print("\t" + "\t".join(parts))
	src port: 58978	dest port: 8899	protocol: 6	impact_flag: 0	blocked: 0
u2spewfoo.py(116):     for row in rows:
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
u2spewfoo.py(117):         parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
 --- modulename: trace, funcname: _unsettrace
trace.py(80):         sys.settrace(None)
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/trace.py", line 819, in <module>
    main()
  File "/usr/lib/python2.7/trace.py", line 807, in main
    t.runctx(code, globs, globs)
  File "/usr/lib/python2.7/trace.py", line 513, in runctx
    exec cmd in globals, locals
  File "bin/idstools-u2spewfoo", line 12, in <module>
    sys.exit(main())
  File "/home//nobackup/build/py-idstools/idstools/scripts/u2spewfoo.py", line 191, in main
    print_record(record)
  File "/home/nobackup/build/py-idstools/idstools/scripts/u2spewfoo.py", line 173, in print_record
    print_event(record)
  File "/home/nobackup/build/py-idstools/idstools/scripts/u2spewfoo.py", line 117, in print_event
    parts = ["%s: %s" % (col[0], col[2](event[col[1]])) for col in row]
KeyError: 'pad2'

Running latest snort-2.9.9.0. Anything I can do to troubleshoot on this end? Running the u2spewfoo that comes with snort runs fine. Thank you.

This app works great with snort.u2 files...I'm wondering if you can integrate appid u2 files as well?

[15:52:26 tester:/opt/snort/var/log$] sudo idstools-u2json --snort-conf /opt/snort/etc/snort.conf -C /opt/snort/etc/classification.config -S /opt/snort/etc/sid-msg.map -G /opt/snort/etc/gen-msg.map --output /tmp/test.json --stdout appid-events-u2.log.1474407059
INFO: Loaded 48171 rule message map entries.
INFO: Loaded 70 classifications.
Traceback (most recent call last):
  File "/usr/local/bin/idstools-u2json", line 12, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/u2json.py", line 318, in main
    for record in reader:
  File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 606, in next
    record = self.reader.next()
  File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 569, in next
    return read_record(self.fileobj)
  File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 539, in read_record
    return decode_record(rtype, buf)
  File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 500, in decode_record
    return DECODERS[record_type].decode(buf)
  File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 347, in decode
    values = struct.unpack(self.format, buf)
struct.error: unpack requires a string argument of length 76

Thank you.

When parsing a merged rule file, using in example rule.parse_file, is not possible to assign the group parameter and the rule group is not parsed into the object.

However the group is somehow standardized in the rule msg using this convention:

So, if rules are well writtent and no group is provided the parser should extract it from the msg second word.

If you find this interesting I can make a PR for this feature.

I'm trying to replicate, kind of, the info that Barnyard2 provides in database tables, but
certain event fields seem to be missing, or I'm not sure how to get them, in idstools.
I'm using:

u2json_reader = unified2.SpoolEventReader(
        directory=args.directory,
        prefix=args.prefix,
        follow=args.follow,
        bookmark=args.bookmark)

In the generator, I print out the event dict as received by:

for event in u2json_reader:
   . . .

... here's a list of event fields I can't find in the output, and
I've referenced the mysql table/column names as used by barnyard2:

signature:
  sig_priority ... is: priority
  sig_class_id ... is: signature-id ?
  sig_class_name
  sig_rev ... is: signature-revision
  sig_sid
  sig_gid

iphdr:
  ip_ver
  ip_hlen
  ip_tos
  ip_len
  ip_id
  ip_flags
  ip_off
  ip_ttl
  ip_proto ... is: proto ?
  ip_csum

tcphdr:
  tcp_seq
  tcp_ack
  tcp_off
  tcp_res
  tcp_flags
  tcp_win
  tcp_csum
  tcp_urp

udphdr: (not sure about these)
  udp_len
  udp_csum

icmphdr:
  icmp_type ... is seen in output
  icmp_code ... is seen in output
  icmp_csum
  icmp_id
  icmp_seq

Suggestions?

Just wondering if I would be missing any important info if I ignore these raw options:

for event_packet in event["packets"]:
  data = event_packet["data"]
  decode_ethernet_data = packet.decode_ethernet(data)
  decode_ethernet_data.pop("tcp_options_raw", None)
  decode_ethernet_data.pop("ip_options_raw", None)
  decode_ethernet_data.pop("ip6_source_raw", None)
  decode_ethernet_data.pop("ip6_destination_raw", None)
  . . .

... aren't these already present in other fields?

I came up with a way to handle these payload fields:
"tcp_payload"
"udp_payload"
"icmp_payload" ... the only one for IPv6?
... so if I can ignore the "_raw" fields above then I will be finished with an open source program to read
unified2 files and index them into elasticsearch
... no barnyard2, no mysql, no ancient snort schema :)

u2eve did not recognise the last event created by snort. u2spewfoo show the last event, so it's successfully updated to the unified2 log file. u2eve only reports the prior event.

When try to process a unified log file with u2json, all I get is the following:
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
Discarding non-event type while not in event context.
...

The command I am running is: sudo idstools-u2json --snort-conf /etc/snort/snort.conf unified2.log
and I know for a fact that data is in that file

Most likely an OE on my part but any help would be greatly appreciated

Rulecat script when using the option --merged always generate a report as you can see in the code, WHICH is logged with level INFO.

If the option -q or --quiet is given this results in an unnecessary execution, because the report is generated but never logged, because the loggin level raises to WARNING.

This code shoul be preceeeded by an if statement that checks if the logging level will permit report display before generating it.

Should zero-sized files like:

-rw------- 1 snort snort        0 Oct 13 17:31 snort.u2.1413221470

cause SpoolEventReader to not all/other unified2 files in the same folder ?

UnicodeDecodeError is caused by raw fields like:
tcp_options_raw
ip_options_raw
ip6_source_raw
ip6_destination_raw
... the error message contains:
UnicodeDecodeError('utf8', "\x01\x01\x08\n's\x04j\x14\xb2{", 10, 11, 'invalid start byte')

This can be re-created, before indexing into elasticsearch, using:
output["packet_details"] = packet.decode_ethernet(data)
from elasticsearch.serializer import JSONSerializer, Deserializer, DEFAULT_SERIALIZERS, TextSerializer
test_serializer = JSONSerializer().dumps(output["packet_details"])

For testing I'm using unified2-current.log and these files:
https://github.com/mephux/unified2/tree/master/example/seeds
... these log files are from 2010/11, so old but not too old, just trying to test as
many new/old unified2 files as I can find.

The current solution is to ignore these fields and not store
them in elasticsearch, but it might be useful to have them
in some format/encoding.

Suggestions ?
Is there some encoding/decoding/formatting of the "*_raw" fields I should do before
trying to index them ?

Here is the end of the run. Rules download properly, but the merged rule file is never updated. If I run with no arguments it grabs the open source rules and does not appear to error out.

Enabled 480 rules for flowbit dependencies.
Recording file /etc/suricata/rules/merged.rules with hash '94bbee8c808b0d5052f8917e51b0749a'.
Traceback (most recent call last):
File "/usr/local/bin/idstools-rulecat", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 752, in main
write_merged(args.merged, rulemap)
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 415, in write_merged
idstools.rule.parse_fileobj(open(filename)))
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/rulecat.py", line 454, in build_rule_map
if rule.id not in rulemap:
File "/usr/local/lib/python2.7/dist-packages/idstools/rule.py", line 158, in id
return (int(self.gid), int(self.sid))
TypeError: int() argument must be a string or a number, not 'NoneType'

Here is my config:

--etpro=
--suricata=/usr/local/bin/suricata
--merged=/etc/suricata/rules/merged.rules
--sid-msg-map-2=/etc/suricata/rules/gen-msg2.map
--sid-msg-map=/etc/suricata/rules/gen-msg.map
--enable=/etc/suricata/rules_enable.conf
--disable=/etc/suricata/rules_disable.conf
--modify=/etc/suricata/rules_modify.conf
--threshold-in=/etc/suricata/threshold.in
--threshold-out=/etc/suricata/threshold.config
--post-hook=sudo kill -USR2 $(cat /var/run/suricata.pid)
--temp=/etc/suricata/rules/raw/
-v

I've verified that all the paths listed above exist. Help is appreciated. Not sure if this is a problem with my version or a problem with the way I've configured it. Used pip to install, idstools (0.5.2)

idstools-rulecat crashes with an exception when using Python 3 and Suricata is installed:

[root@9d3d1f76ead0 ~]# idstools-rulecat 
Traceback (most recent call last):
  File "/usr/bin/idstools-rulecat", line 12, in <module>
    sys.exit(main())
  File "/usr/lib/python3.4/site-packages/idstools/scripts/rulecat.py", line 725, in main
    args.url.append(resolve_etopen_url(args.suricata))
  File "/usr/lib/python3.4/site-packages/idstools/scripts/rulecat.py", line 642, in resolve_etopen_url
    suricata_version = idstools.suricata.get_version(suricata_path)
  File "/usr/lib/python3.4/site-packages/idstools/suricata.py", line 61, in get_version
    m = re.search("version ((\d+)\.(\d+)\.?(\d+|\w+)?)", output.strip())
  File "/usr/lib64/python3.4/re.py", line 170, in search
    return _compile(pattern, flags).search(string)
TypeError: can't use a string pattern on a bytes-like object

The error is in parsing the output of "suricata -V".

Is that possible? Say I have u2eve reading two separate u2 files from two different snort instances....can I write to just one json file? Or do I have to write to 2 separate ones? Thanks Jason!

According to the docs https://www.snort.org/faq/readme-unified2/ and http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node44.html the Extra Data record only supports 24 bytes where your EXTRA_DATA_FIELDS enum contains 32, the first two fields (event-type, event-length) do not appear to be documented. As these two sites differ in portions referencing the same events it could be a documentation issue and not your code.

This request is divided in two step:

1. Snort community rules are not handled properly:

$ idstools-rulecat --url "https://www.snort.org/rules/community" --merged community.rules
2017-05-16 17:51:56,704 - <INFO> -- Found Suricata version 3.1.2 at /usr/bin/suricata.
2017-05-16 17:51:56,705 - <INFO> -- Checking https://www.snort.org/rules/community.md5.
2017-05-16 17:51:57,674 - <ERROR> -- Failed to check remote checksum: HTTP Error 406: Not Acceptable
2017-05-16 17:51:57,675 - <INFO> -- Fetching https://www.snort.org/rules/community.
 100% - 295175/295175                 
2017-05-16 17:51:59,439 - <INFO> -- Done.
2017-05-16 17:51:59,439 - <INFO> -- Loaded 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Disabled 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Enabled 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Modified 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Dropped 0 rules.
2017-05-16 17:51:59,440 - <INFO> -- Enabled 0 rules for flowbit dependencies.
2017-05-16 17:51:59,440 - <INFO> -- Writing community.rules: added: 0; removed 0; modified: 0

But if I manually download the file from my browser what I get is a .tar archive which contains a community.rules with about 3500 rules.

2. Add parameters like ET also for snort

With the latest version (master) also snort registered rules download is supperted via the --url parameter:

idstools-rulecat --url "https://www.snort.org/rules/snortrules-snapshot-2976.tar.gz?oinkcode=<onikcode>"

What I propose is to add a couple of arguments to handle snort like ET:

--snort-version                 Snort version (choose from 2976, 2983, 2990)
--snort-community               Download snort community rules
--snort-registered <oink-code>  Download snort registered rules providing your oink-code

The URLs for snort rules are:

# snort community
https://www.snort.org/rules/community

# snort registered
https://www.snort.org/rules/snortrules-snapshot-<snort-version>.tar.gz?oinkcode=<oinkcode>

The Rule object has no ruleset attribute.

Examples:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET MALWARE Windows executable sent when remote host claims to send an image 2"; flow: established,from_server; content:"|0d 0a|Content-Type|3a| image/jpeg|0d 0a 0d 0a|MZ"; fast_pattern:12,20; classtype:trojan-activity; sid:2020757; rev:2;)

is from ET ruleset

alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)

is from SNORT GPL

Usually the ruleset is the first word in the Rule.msg attribute, so it should not be difficult to extract. If someone does not use this convention it should be possible to opt-out this parsing function, maybe letting the parser get this value as an optional argument.

I would like to request a drop.conf for changing all/some/specified "alert" rules to "drop" rules.

Which is the format of the 'data' field that provides SpoolEventReader? Is the same for every reader. Ii is no the hexadecimal and it seems that the library add some content.
Here is an extract example:
\x00!\xd7j\xe4\x00\xdcJ>\x88*R\x08\x00E\xc0\x00q\xf4

I think:
"unified2.log", tail=True)
in the README.rst should be:
"unified2.log", follow=True)
... not really as issue :)

idstools-u2json --output test.json --packets --extra-data snort2.log.1393411837
WARNING: No alert message map entries loaded.
WARNING: No classifications loaded.
Traceback (most recent call last):
File "/usr/local/bin/idstools-u2json", line 5, in
pkg_resources.run_script('idstools==0.5.0dev', 'idstools-u2json')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/idstools-0.5.0dev-py2.7.egg/EGG-INFO/scripts/idstools-u2json", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools-0.5.0dev-py2.7.egg/idstools/scripts/u2json.py", line 374, in main
encoded = json.dumps(output_filter.filter(event))
File "/usr/lib/python2.7/json/init.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd8 in position 272: invalid continuation byte

Possible solution to base64 encode data field:

Change from line 186:
output_data = {
"data-type": data["data-type"],
"event-type": data["event-type"],
"type": data["type"],
"data": data["data"]
}
to
output_data = {
"data-type": data["data-type"],
"event-type": data["event-type"],
"type": data["type"],
"data": base64.b64encode(data["data"])
}

But not in its raw format, make it usable.

I'm trying to get at what analysts call "payload" data using idstools-u2json. I tried using the master branch with the additional params of --packets --extra-data, and I never see anything in extra-data field, but I do see what looks like binary data in the 'packet'(s) field.

When using Barnyard2 it places the payload data, if any, in the mysql/db table called 'data', which is represented in the column 'data_payload' as a hex string ... it would be nice to store this as both
hex and ascii in elasticsearch.

Also, I can't find the sensor(sid), hostname, or interface fields ... as sometimes our snort events are gathered from multiple sensors.

Suggestions ?
... apologies if this is the wrong place to ask a question as this isn't a bug/fix

root@snort1:/etc/snort# idstools-u2json -C /etc/snort/classification.config -S /etc/snort/sid-msg.map -G /etc/snort/gen-msg.map /var/log/snort/snort.log.1454584880
INFO: Loaded 514 rule message map entries.
INFO: Loaded 38 classifications.
Traceback (most recent call last):
File "/usr/local/bin/idstools-u2json", line 12, in
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/idstools/scripts/u2json.py", line 308, in main
for record in reader:
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 586, in next
record = self.reader.next()
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 549, in next
return read_record(self.fileobj)
File "/usr/local/lib/python2.7/dist-packages/idstools/unified2.py", line 522, in read_record
raise err
EOFError

CentOS7, python 3.4.5 - have a problem with

Failed to encode record as JSON: __init__() got an unexpected keyword argument 'encoding':

but looking at https://docs.python.org/3/library/json.html#basic-usage json.dumps does not have the encoding argument, and encoding in json.loads is deprecated.

HI there,

first of all thanks for this library, it is very good and helpful!

However either I'm doing something wrong or there is a problem with bookmarking.

I'm working with version 0.3.1 and have following (simplified) code. For debugging purposes this I was not using a for statement:

bookmark_filename, bookmark_offset = json.load(open(config['ids']['bookmark_file']))
self.ids_reader = unified2.SpoolEventReader(
                                config['ids']['ids_log_dir'],
                                config['ids']['ids_log_prefix'],
                                init_filename=bookmark_filename,
                                init_offset=bookmark_offset)

event = self.ids_reader.next()
logging.log(logging.DEBUG, "Processing new event %d from sensor %d" % (event['event-id'], event['sensor-id']))
json.dump(self.ids_reader.tell(), open(config['ids']['bookmark_file'], "w"))

So what I've expected is that if I run this piece of code multiple times, it accesses each event in the the specified files. However if I look in the logging output I see that there is a jump over one record:

2014-04-16 17:30:30,957 [INFO    ] Initializing unified2.SpoolEventReader
2014-04-16 17:30:30,969 [INFO    ] Start processing new incoming and unprocessed old events
2014-04-16 17:30:30,973 [WARNING ] Discarding non-event type while not in event context.
2014-04-16 17:30:30,977 [DEBUG   ] Processing new event 106 from sensor 0
2014-04-16 17:31:42,452 [INFO    ] Initializing unified2.SpoolEventReader
2014-04-16 17:31:42,453 [INFO    ] Start processing new incoming and unprocessed old events
2014-04-16 17:31:42,454 [WARNING ] Discarding non-event type while not in event context.
2014-04-16 17:31:42,455 [DEBUG   ] Processing new event 108 from sensor 0
2014-04-16 17:31:44,677 [INFO    ] Initializing unified2.SpoolEventReader
2014-04-16 17:31:44,677 [INFO    ] Start processing new incoming and unprocessed old events
2014-04-16 17:31:44,678 [WARNING ] Discarding non-event type while not in event context.
2014-04-16 17:31:44,678 [DEBUG   ] Processing new event 110 from sensor 0

It seems that this warning indicates that something went wrong while parsing the file. Did I do anything wrong or is there an issue with the bookmarking?

Note: If I parse the file without the bookmark I can access all records.

Thanks

based on my checking on unified2.py the EVENT_IP6_FIELDS should be fixed:

Field("source-ip.raw", 4, "16s"),   --->    Field("source-ip.raw", 16, "16s"),
Field("destination-ip.raw", 4, "16s"),    --->    Field("destination-ip.raw", 16, "16s"), 

otherwise these errors will be return:
"struct.error: unpack requires a bytes object of length 76" on line 343

format is '>LLLLLLLLL16s16sHHBBBB' and len(buf) should be 76 but self.fixed_len is 52 now

When reading a unified2 log from Snort, the decode_record function fails with the following error:

struct.error: unpack requires a bytes object of length 84

The EVENT_IP6_FIELDS tuple tells the decoder to look for only 64 bytes. I am not sure if this is because of the sensor I used or because Snort has changed the signature. I fixed the bug by replacing

Field("source-ip.raw", 4, "16s"), Field("destination-ip.raw", 4, "16s")

with,

Field("source-ip.raw", 16, "16s"), Field("destination-ip.raw", 16, "16s")

Rule:

alert tcp 93.174.88.0/21 any -> $HOME_NET any (msg:"SN: Inbound TCP traffic from suspect network (AS29073 - NL)"; flags:S; reference:url,https://suspect-networks.io/networks/cidr/13/; threshold: type limit, track by_dst, seconds 30, count 1; classtype:misc-attack; sid:71918985; rev:1;)

fast hit:

05/02-16:18:54.125546  [**] [1:71918985:1] SN: Inbound TCP traffic from suspect network (AS29073 - NL) [**] [Classification: Misc Attack] [Priority: 2] {TCP} 93.174.95.106:40082 -> x.x.x.x:2222

json hit from the unified file:

{"timestamp": "2017-05-02T16:18:54.125546+0000", "sensor_id": 0, "event_id": 1738, "event_second": 1493741934, "event_type": "alert", "src_ip": "93.174.95.106", "src_port": 40082, "dest_ip": "x.x.x.x", "dest_port": 2222, "proto": "TCP", "flow_id": 5007401709, "alert": {"action": "allowed", "gid": 1, "signature_id": 71918985, "rev": 1, "signature": null, "category": "Misc Attack", "severity": 2}, "packet": "fA7<snip>AAAAAAAAAAA==", "packet_printable": "|...H.0.\r\n........E(.(.v..q.2.]._j...C....;.......P.............", "packet_info": {"linktype": 1}}

It appears that the ids_alert field is missing when these hit.

The way unified2 logging works, the packets (or any non-event record) following an event record may not be associated with that event, but an event that happened before it.

Currently the record aggregator will bundle all non-event records with the most recent event until a new event record is received. Instead it will need to verify that the record does belong to the event that is currently being aggregated.

This also means that an Event object may not contain all the packet records associated it with it, as there may be an event record in the unified2 log file before a packet record for the previous event is seen.

I think the only solution is that the EventReader's make a best effort to gather all the records together in an event. Perhaps the aggregator could have a timeout where it holds onto multiple events for an amount of time waiting for more records, but there could still be a record that appears outside that window.

Still needs some thought.

This fails:

python2 /usr/bin/idstools-eve2pcap merged.log.eve -o merged.log.eve2pcap --dlt RAW
python3 /usr/bin/idstools-eve2pcap merged.log.eve -o merged.log.eve2pcap --dlt RAW

ValueError: time data '2017-01-02T09:51:30.644790+0100' does not match format '%Y-%m-%dT%H:%M:%S.%f-0600'

with the following merged.log.eve:

{"timestamp": "2017-01-02T09:51:30.644790+0100", "sensor_id": 0, "event_type": "alert", "src_ip": "10.255.2.100", "src_port": 50621, "dest_ip": "10.255.2.160", "dest_port": 80, "proto": "TCP", "flow_id": 100663492, "alert": {"action": "allowed", "gid": 1, "signature_id": 10000002, "rev": 1, "signature": null, "category": null, "severity": 0}, "packet": "RQAAPBCRQABABg8qCv8CZAr/AqDFvQBQfykbqgAAAACgAnIQTs0AAAIEBbQEAggKAAULOwAAAAABAwMH"}

The problem is in

On python >= 3.2 one could do:

dt = datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%f%z")

but for both python2 and python3 maybe this?

from dateutil import parser
dt = parser.parse(timestamp)

rpmlint (a tool for checking RPM packaging style) uses shebang to decide whether a python file is a script.

https://github.com/jasonish/py-idstools/tree/1dccb619555097cbbddb60095fd8f7da385c722a/idstools/scripts packages the following files as non-executable scripts under the python modules directory:

idstools/scripts/u2fast.py
idstools/scripts/u2spewfoo.py
idstools/scripts/u2eve.py
idstools/scripts/u2json.py
idstools/scripts/eve2pcap.py
idstools/scripts/rulecat.py
idstools/scripts/gensidmsgmap.py

The recommended handling, as reported by rpmlint is to remove the shebang "#! /usr/bin/env python" from those files, like in the case of idstools/scripts/dumpdynamicrules.py

Unifiedbeats, which is just what I was looking for, is no longer being maintained. My hope and desire is to use idstools to continually read u2 files and send all the data to a listening elasticsearch instance. The bonus with u2 is the extradata and actual packet fields, otherwise just using rsyslog to read the fast file is more all I'd need. Is there a way I can achieve this with idstools as they are now? Thank you.

When I look at the barnyard2 data in the mysql table "sensor" the value for
"sid" is 1, but the value of "sensor_id" from u2json.py is always 0.

u2json
-C=/my/classification.config
-S=/my/sid-msg.map
-G=/my/gen-msg.map
--directory=/my
--prefix=snort.unified2
--follow
--bookmark
--packets
--extra-data

I think it should be 1, or am I missing something
... or is this a "hard coded" setting based on the directory of snort logs ?
Suggestions ?

I'm trying to use idstools.rule.parse to validate rules, and this one has an invalid direction:

idstools.rule.parse("""#alert tcp $HOME_NET any >>> $EXTERNAL_NET $HTTP_PORTS ...)

but the rule is returned as valid
... I think the only valid directions are '<>', '<-', '->'
... everything else seems to work for validation

Instead mask it as or something.

cd /tmp
git clone https://github.com/jasonish/py-idstools
cd py-idstools
git rev-parse HEAD
#6821df33bfaaf39331cf27843144d48cc3f9b8dc
PYTHONPATH=. python3 bin/idstools-rulecat -o /etc/suricata/rules
2017-05-18 11:20:43,213 - <INFO> -- Found Suricata version 3.2.1 at /usr/sbin/suricata.
Traceback (most recent call last):
  File "bin/idstools-rulecat", line 12, in <module>
    sys.exit(main())
  File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 766, in main
    files = Fetch(args).run()
  File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 284, in run
    files.update(self.fetch(url))
  File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 262, in fetch
    tmp_filename = self.get_tmp_filename(url)
  File "/tmp/py-idstools/idstools/scripts/rulecat.py", line 251, in get_tmp_filename
    url_hash = hashlib.md5(url).hexdigest()
TypeError: Unicode-objects must be encoded before hashing

See https://bugzilla.redhat.com/show_bug.cgi?id=1452025

I have the following in my disable.conf file:

re:.*
re:^#.*
re:ET.* TOR
re:ET.* SCADA
re:ET.* SCADA_SPECIAL
re:ET.* DELETED
re:ET.* SURICATA STREAM

and some individual rule IDs. Then, in my enable.conf file I have a few entries like this:

re:^[^#].* EXPLOIT .*
re:^[^#].* CINS .*
re:^[^#].* CURRENT_EVENTS .*
and some other individual rule IDs.

I am still getting tons of other rules enabled, such as many of the ET DELETED rules, etc. Can you please explain the order of precedence, etc.? Does disable override enable, modify, etc.? Is there a way for me to tell rulecat to favor disable over enable?

If a semicolon (;) is present (escaped, obviously) in the rule msg option the parser gets confused.

Example:

$ my_ryle = '''alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"example of semicolon\; usage here"; classtype:trojan-activity; sid:9999; rev:1;)'''
$ parsed_rule = rule.parse(my_rule)
$ print parsed_rule.msg
'"example of semicolon\\'
$ print parsed_rule.options
[{'name': 'msg', 'value': '"example of semicolon\\'}, {'name': 'usage here"', 'value': None}, {'name': 'classtype', 'value': 'trojan-activity'}, {'name': 'sid', 'value': '9999'}, {'name': 'rev', 'value': '1'}]

Some rules (ETPRO) really use semicolon in the msg option.

in EVENT_FIELDS and EVENT_IP6_FIELDS in unified2.py, you have the "sport-itype" and "dport-icode" fields. These should be four fields instead of two:
sport -> source Port
dport -> destination port

itype -> ICMP Type
icode -> ICMP code

I'm probably missing something simple, but this was working before ... but now:

>>> import idstools
>>> from idstools import packet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name packet

with:
python 2.7.6
pip freeze idstools==0.4.4 ... this worked using 0.5.0dev

Please ignore the above, I forgot that you have to do:
pip install git+https://github.com/jasonish/py-idstools.git
and not:
pip install idstools
... but maybe this will help others that make the same mistake.