sekoia-io / intake-formats Goto Github PK

Hi!

This event (anonymized from a real one) causes a parsing error in the Cloudflare Access Requests intake, with the message
set:set_cloudflare_fields:cloudflare - isn't defined in taxonomy.

{"Action":"sso","Allowed":true,"AppDomain":"foo-bar-baz.xyz","AppUUID":"f1c47079-d821-3e7a-8f67-625c1e4fb6a4","Connection":"onetimepin","Country":"fr","CreatedAt":"2023-04-27T14:40:10Z","Email":"[email protected]","IPAddress":"78.101.123.45","PurposeJustificationPrompt":"why?","PurposeJustificationResponse":"because!","RayID":"7bf00f0ea9e9f858","TemporaryAccessApprovers":["[email protected]"],"TemporaryAccessDuration":900,"UserUID":"03251629-69f1-5ba0-a154-463be885554b"}

On intake 0d1009c7-8c78-476a-8c1c-2439ddb57462, the DNS query logs are not parsed correctly.

The problem seems to be that the syslog message starts with queries: client , e.g.:

queries: client @0x7f62b80115d0 192.168.0.2#55473 (docs.sekoia.io): query: docs.sekoia.io IN AAAA + (192.168.0.1)

whereas the parser expects the message to start with client :

According to the Infoblox docs, the name of the log category (here queries: ) can be prefixed or not to the syslog messages depending on the configuration.

Logging Category: Select one of the following logging categories:

• Send all: [...] the syslog messages are not prefixed when you select this option.
• Send selected categories: [...] The syslog messages are prefixed with a category name to which it belongs. [...]

The integration documentation doesn't currently specify either option.

I think that we should modify the parser to handle both cases.

Here's how the official Elastic integration seems to parse the logs:

^(%{NOTSPACE:infoblox_nios.log.dns.category}:)?\\s*client %{DATA} %{IP:client.ip}...

CI testing for smart descriptions

• Create python test unit for smart description
• Add CI check for smart description

In CrowdStrike IDP events, some timestamp are not in the UNIX format but in the LDAP format.

They are currently not converted correctly and events are inserted several years in the past because of this.

For an example, look for events with crowdstrike.event_type: IdpDetectionSummaryEvent.

If useful, here's how the official Elastic parser handles the conversoin:
https://github.com/elastic/integrations/blob/main/packages/crowdstrike/data_stream/falcon/elasticsearch/ingest_pipeline/default.yml#L38-L80

Notes:

• some message ids don't have brackets
• some events seem to have multiple messages (message is an array), not sure why