The standard rendering of the ID (eg 1:ZEYOYMeyZNQC9DAdgsBZCtiTKqw=) is not only not very nice to look at, but can break standard string handling in SIEM pipelines and other tools (so you need to add quotes, for example). A reduced character set would work better here, at the expense of a slightly longer string.

I'm considering moving to something like base58 as a default for version 2 of the ID. Thoughts welcome.

Adding test vectors in the default directory of test pcap to ensure implementation is correct.

If there was a 4 tuple hash, then I could share these hashes with other people and tools, between different networks, and use them in very much the same way. Dropping the source address would mean that hash x can be applied against traffic in any network inside and outside of a particular organization. it would put the community in community ID.

The Community ID could include features beyond the flow tuple, such as the presence of particular file transfers in the flow. This could aid in disambiguation of flows with otherwise colliding IDs, but narrows applicability to monitors that are able to track at this level of inspection.

This is a feature suggestion from SuriCon 2018.

Is mapper entry

 dpkt.icmp.ICMP_RTRADVERT:      dpkt.icmp.ICMP_RTRSOLICIT,

missing?

A few folks have suggested that one could share the ID as an anonymous/pseudonymous substitute for the flow tuple, to avoid revealing the actual flow. (In analogy to sharing a file hash instead of the actual file, for example.) Applicability here seems much more narrow since the ID would most likely be of value only to the parties able to observe the underlying flow.

It may be of interest to keep the flow endpoints discernible in the ID (as a pair of hashes, perhaps) — doing so would allow checking whether one has also seen a certain endpoint in abusive behavior, etc. But that immediately leads to separating the address from the port, so we're essentially down to rendering each part of the flow tuple separately. Seems in those settings you might as well not use the ID in the first place.

I'm afraid I don't remember all individuals who have brought this up. — @vivekrj asked on Twitter, as did one participant at the 2018 Bro workshop in Karlsruhe, Germany.

Additional thoughts are very welcome.

Hey,

While doing some research on various data we have, we've seen across two separate applications which saw the same flow, they each had their own interpretation of src/dst ordering.

You'll have to excuse the formatting but in application 1 (the data is fake but the example has happened), the flow was seen as:

src,srcp,dst,dstp
aaa,65000,bbb,443

While in application 2 it was reversed:

src,srcp,dst,dstp
bbb,443,aaa,65000

Looking at the spec itself, we appear to leave the ordering of the src/dst combination up to the individual application and if we had implemented community ID in both applications, we would have ended up with a different community ID for the same flow.

I would suggest that the spec is updated to specify how the src/dst combination should be calculated/sorted in order to ensure a consistent community ID is generated for the same flow (regardless of what the underlying application thinks the direction is).

Thanks,
Conor

We've built community ID support into VAST to allow for pivoting between ingested PCAPs, Suricata, Zeek, and NetFlow/IPFIX. Our C++ implementation of community ID computation is available here.

We have observed approximately 8% loss in IPFIX ingestion performance when enabling community ID computation.

Recently we have experimented with replacing SHA1 in the community ID computation, and have had great results using xxHash. The overall performance loss went down from 8% to 3%.

I am proposing to use XXH3 (supposed to be stabilized in H1 2020) for community ID v2 to improve the usability on high-throughput paths.

The time dimension is currently completely missing from the ID computation, because obvious ways of incorporating it (e.g. by rounding to the current day) impose risks of ID deviation around the rounding boundary when multiple monitors aren't perfectly in sync. Suggestions for techniques not prone to this shortcoming are very welcome.

One such suggestion is the use of ssdeep-style fuzzy hashing. I don't think it immediately applies because time representations are subject to major jumps themselves — consider a counter that jumps from 199,999 to 200,000. But here too, discussion is very welcome.

This was a talk discussion question at SuriCon 2018.

I reviewed this file and ran a test:

https://github.com/corelight/community-id-spec/blob/master/baseline/baseline_nob64.json

The last two test:
{"proto": 46, "saddr": "10.1.12.1", "daddr": "10.1.12.2", "sport": null, "dport": null, "communityid": "1:28794b920a095bb89f513c934a0c957e4147ccac"},
{"proto": 46, "saddr": "10.1.12.2", "daddr": "10.1.12.1", "sport": null, "dport": null, "communityid": "1:28794b920a095bb89f513c934a0c957e4147ccac"}

The community ID for this appears to be incorrect. I think this community ID came from having the IPs flipped based on my test. When I flip it so 10.1.12.2 comes first and 10.1.12.1 comes second in the byte array it gives me: 28794b920a095bb89f513c934a0c957e4147ccac

I think it should be:
1f2f15b2c1f825376e9a372350b6e06202e7f1c7

Is this script Python2 only? If so, can it be updated to support Python3?

$ python3 community-id.py --verbose test.1684238207.pcap
Traceback (most recent call last):
  File "/opt/pcap/community-id.py", line 381, in <module>
    sys.exit(main())
  File "/opt/pcap/community-id.py", line 376, in main
    hasher.pcap_handle(pcapfile)
  File "/opt/pcap/community-id.py", line 158, in pcap_handle
    self._packet_handle(tstamp, pktdata)
  File "/opt/pcap/community-id.py", line 358, in _packet_handle
    print_result(tstamp, pkt, self._packet_get_comm_id(pkt, key))
  File "/opt/pcap/community-id.py", line 302, in _packet_get_comm_id
    dlen = hash_update(struct.pack('!H', self.comm_id_seed)) # 2-byte seed
  File "/opt/pcap/community-id.py", line 297, in hash_update
    hexbytes = ':'.join('%02x' % ord(b) for b in data)
  File "/opt/pcap/community-id.py", line 297, in <genexpr>
    hexbytes = ':'.join('%02x' % ord(b) for b in data)
TypeError: ord() expected string of length 1, but int found

Here test.1684238207.pcap is the output of a 1-second tcpdump and a simple HTTP request across the wire.

IP addresses must be compared to form community IDs.

Is 192.0.2.77 greater than ::1 or less than it? One way to compare them is to upcast IPv4 address to an IPv6 address by using the IPv4-mapped range (i.e. ::FFFF:192.0.2.77). Is this what implementations are expected to do?

Yes. The Community ID can be computed on arbitrary data, not necessarily based on raw packets. Some concerns apply — for example, the spec requires network byte order for some hash inputs. Assuming such functionality is supported any data store could compute the ID.

Also see #5.

This was a talk discussion question at SuriCon 2018.

I'm suspect this is a typo:

function community_id_icmp(int seed, ipaddr saddr, ipaddr daddr, int type, int code, int seed=0)

DJ Gregor suggested one could simply append the flow's timestamp to the end of the ID when wanting to filter out clashing, unrelated flows. Doing so would allow you to prefix-match on the resulting overall string, which somewhat isolates you from the clock-rollaround problems naive inclusion of timestamps would have.

A given network flow's 5-tuple will differ depending on whether it's perceived internally, externally, before or after a NAT, etc. Can the Community ID accommodate this?

The short answer is no, since there's an assumption that the ID is based on the observed flow tuple, so presence of a NAT will make a semantically identical flow look like two different ones. However, this relates to #4 in that the ID could be based on other flow properties such as QUIC's connection ID.

This was a talk discussion question at SuriCon 2018.

At https://tools.ietf.org/html/rfc8335 an additional request-reply pair is documented.
ICMP field Type: Extended Echo Request. The value for ICMPv4 is 42. The value for ICMPv6 is 160
ICMP field Type: Extended Echo Reply. The value for ICMPv4 is 43. The value for ICMPv6 is 161

Since that wasn’t in the version 1 table of the reference Corelight python, hashes using that code would set one or the other "port-like value" to the code octet instead of to the type octet of the corresponding request-response peer. This issue was also reported as: corelight/pycommunityid#2 (comment)

This specific case also highlights that: which pairs are in the table of the corresponding request-response ICMP and ICMPv6 pairs, is a specification point, and should not be subject to variation, or derived from a reference implementation outside of the specification.

This implementation looks great and I'd love to use it in my app. Do you give people permission to use the code? If so, can you add an open source license to the project? I see you have the BSD license on some of the other repos. Adding it to this one would give me permission to use this code.
Thanks!

Implemented the algorithm and tested the resulting ids comparing against this repo's python script and also against wireshark's community id analysis tool. My implementation's results seemed to always match wireshark's but not corelight's.

When having same ip addresses for src and dst, the following port values seem to be handled differently by corelight's script:

     ip_src   ip_dest  proto(tcp)  port_src  port_dst  community id from coreligh python script   community id from wireshark
     1.2.3.4  1.2.3.4      6         3344      1122     1:CY1T7/6B7r9W3LMnzSws9RXqqbQ=             1:3seqIXu+5y8sFuE3lLtWR/KnSWo= 
     1.2.3.4  1.2.3.4      6         1122      3344     1:CY1T7/6B7r9W3LMnzSws9RXqqbQ=             1:3seqIXu+5y8sFuE3lLtWR/KnSWo=  
     1.2.3.1  1.2.3.1      6         3344      1122     1:jD1eCyop8ZzeL/0xgO58JtpHPLE=             1:GfxcHO3Gsn1cQxpPJoBhxUMcrbU= 
     1.2.3.4  1.2.3.4      6         5566      7788     1:YZWovxgLMFDntXoQs0LgEGs9QcQ=             1:BjNbSLaYeZZX0M1egqjh1Akg9yw=   
     1.2.3.4  1.2.3.4      6         6655      8877     1:7hdcx4YnNvllNYNSbtSzegQFnjg=             1:miYC8NFsg/sTi5HwWjjyifbbp+8=   

Port combinations like: {1122, 44424}, {334, 44424}, {1133, 2244}, {8899, 1199} didn't seem to cause any issues though.

In function is_ordered(), changing port comparison in the return to:

int.from_bytes(nbo.sport, 'little') < int.from_bytes(nbo.dport, 'little')

seems to get the cases I tried to match.

Absolutely. Suricata and Zeek are initial applications to support the ID, but wider adoption (e.g. in Snort, Wireshark, or Scapy) is explicitly encouraged. Just be aware that the implementation is likely to go substantial updates over time.

This was a talk discussion question at SuriCon 2018.

It doesn't matter, because the input to the hash function is always the binary representation of an address (or port), not a potentially shortened ASCII representation.

This was a talk discussion question at SuriCon 2018.

At my company, we are enriching traffic logs with community ids with an implementation of this spec that we rolled on our own. It would be nice if the spec included some examples of an input 5-tuple along with the expected community id. Then it would be easy to tell if we implemented the spec correctly.

I am writing a program in TypeScript to use the community ID. I have the IPV4 working great, but not so with the IPV6.
I know the python packing runs and works, but is it correct?

I tried to use the verbose mode in the community-id.py to look at the buffer to validate this but it is broken. python community-id.py --no-base64 --verbose pcaps\ipv6.pcap

Your output produces the following:

1500000101.216315 | 1:fea15a78047e8057b52988cccd50ec32ffb0814e | 2607:f8b0:400c:c03::1a 2001:470:e5bf:dead:4957:2174:e82c:4887 6 25 63943
1500000101.416173 | 1:fea15a78047e8057b52988cccd50ec32ffb0814e | 2001:470:e5bf:dead:4957:2174:e82c:4887 2607:f8b0:400c:c03::1a 6 63943 25

My output with my buffer:
expanded and removed ":" from ip.
r1: 2607f8b0400c0c03000000000000001a
r2: 20010470e5bfdead49572174e82c4887
ArrayBuffer {
[Uint8Contents]: <00 00 26 07 f8 b0 40 0c 0c 03 00 00 00 00 00 00 00 1a 20 01 04 70 e5 bf de ad 49 57 21 74 e8 2c 48 87 06 00 00 19 f9 c7>,
byteLength: 40
}
f7aae5da871859418931a2ddfbf7220ecbc50412