Issues uploading Threat Indicators to Sentinel/Defender ATP about connectors HOT 13 OPEN

Hi @blockanz, it seems that there is another environment variable missing in your yml, given the error I would say :

      - RESOURCE_URL=https://graph.microsoft.com
      - REQUEST_URL=/beta/security/tiIndicators

Here is the link to docker-compose with all the environment variables for sentinel, can you compare it with the one you have ?
I hope this solves your problem.

from connectors.

Grrr. I somehow mistyped things and had the INCIDENT_URL with the REQUEST_URL value. Thanks you. I'll test this with the proper values and advise.

from connectors.

@The-Stuke I saw that you created the connector. Do you know what's happening?
Otherwise @Megafredo or @helene-nguyen could you have a look when you have time? This is a connector being under the community supervision FYI

from connectors.

Hi @blockanz, this error occurs when the environment variable "EXPIRE_TIME" is either missing or empty, can you check this variable in your .yml ?
By default in the README:
EXPIRE_TIME=30

from connectors.

@Megafredo

I made the change and added EXPIRE_TIME=30.

Now I get the following errors:

{"log":"{"timestamp": "2024-04-25T01:21:59.424348Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Failed processing data {can only concatenate str (not \"NoneType\") to str}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.42478946Z"}
{"log":"{"timestamp": "2024-04-25T01:21:59.425073Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Message data {{\"version\":\"4\",\"type\":\"create\",\"scope\":\"external\",\"message\":\"creates a IPv4-Addr 123.14.18.239\",\"origin\":{\"socket\":\"query\",\"ip\":\"::ffff:192.168.48.1\",\"user_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"group_ids\":[\"576aa993-0257-46cf-844d-8d5a44128257\"],\"organization_ids\":[],\"user_metadata\":{},\"applicant_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"call_retry_number\":\"1\"},\"data\":{\"id\":\"ipv4-addr--21075343-2f26-5461-9993-263f210858ff\",\"spec_version\":\"2.1\",\"type\":\"ipv4-addr\",\"extensions\":{\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\":{\"extension_type\":\"property-extension\",\"id\":\"94ce7581-6907-41e3-a065-0c9a27bfba74\",\"type\":\"IPv4-Addr\",\"created_at\":\"2024-04-25T01:21:58.817Z\",\"updated_at\":\"2024-04-25T01:21:58.817Z\",\"is_inferred\":false,\"creator_ids\":[\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\"],\"labels_ids\":[\"c13f46fe-addf-4d20-9907-dbc599753220\",\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\"],\"created_by_ref_id\":\"9faf421d-5355-41d9-8731-7f63dc0509ca\"},\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\":{\"extension_type\":\"property-extension\",\"labels\":[\"elf\",\"mozi\"],\"description\":\"Malware payload delivery host\",\"score\":60,\"created_by_ref\":\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\"}},\"object_marking_refs\":[\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\"],\"value\":\"123.14.18.239\"}}}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.42533194Z"}
{"log":"{"timestamp": "2024-04-25T01:21:59.879506Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Failed processing data {can only concatenate str (not \"NoneType\") to str}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.880151227Z"}
{"log":"{"timestamp": "2024-04-25T01:21:59.881081Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Message data {{\"version\":\"4\",\"type\":\"create\",\"scope\":\"external\",\"message\":\"creates a IPv4-Addr 123.14.251.202\",\"origin\":{\"socket\":\"query\",\"ip\":\"::ffff:192.168.48.1\",\"user_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"group_ids\":[\"576aa993-0257-46cf-844d-8d5a44128257\"],\"organization_ids\":[],\"user_metadata\":{},\"applicant_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"call_retry_number\":\"1\"},\"data\":{\"id\":\"ipv4-addr--305c4cae-d829-5ee5-a850-c8fe145146a1\",\"spec_version\":\"2.1\",\"type\":\"ipv4-addr\",\"extensions\":{\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\":{\"extension_type\":\"property-extension\",\"id\":\"cf824fcc-d364-499c-8311-f5e9e3e84126\",\"type\":\"IPv4-Addr\",\"created_at\":\"2024-04-25T01:21:59.351Z\",\"updated_at\":\"2024-04-25T01:21:59.351Z\",\"is_inferred\":false,\"creator_ids\":[\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\"],\"labels_ids\":[\"c13f46fe-addf-4d20-9907-dbc599753220\",\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\"],\"created_by_ref_id\":\"9faf421d-5355-41d9-8731-7f63dc0509ca\"},\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\":{\"extension_type\":\"property-extension\",\"labels\":[\"elf\",\"mozi\"],\"description\":\"Malware payload delivery host\",\"score\":60,\"created_by_ref\":\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\"}},\"object_marking_refs\":[\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\"],\"value\":\"123.14.251.202\"}}}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.881292975Z"}
{"log":"{"timestamp": "2024-04-25T01:22:00.162560Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Failed processing data {can only concatenate str (not \"NoneType\") to str}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:22:00.162989455Z"}
{"log":"{"timestamp": "2024-04-25T01:22:00.163477Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Message data {{\"version\":\"4\",\"type\":\"create\",\"scope\":\"external\",\"message\":\"creates a IPv4-Addr 123.14.252.72\",\"origin\":{\"socket\":\"query\",\"ip\":\"::ffff:192.168.48.1\",\"user_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"group_ids\":[\"576aa993-0257-46cf-844d-8d5a44128257\"],\"organization_ids\":[],\"user_metadata\":{},\"applicant_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"call_retry_number\":\"1\"},\"data\":{\"id\":\"ipv4-addr--d11fbddd-56a6-5f3a-ac93-0456a333fcd6\",\"spec_version\":\"2.1\",\"type\":\"ipv4-addr\",\"extensions\":{\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\":{\"extension_type\":\"property-extension\",\"id\":\"04dd40dc-6b01-46e4-9c37-bc511669cd10\",\"type\":\"IPv4-Addr\",\"created_at\":\"2024-04-25T01:21:59.435Z\",\"updated_at\":\"2024-04-25T01:21:59.435Z\",\"is_inferred\":false,\"creator_ids\":[\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\"],\"labels_ids\":[\"4491d7c7-5744-408e-aa4b-837dd2dd172d\",\"c13f46fe-addf-4d20-9907-dbc599753220\",\"42c9846a-d05b-4bf4-9956-236dfdae90e6\",\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\"],\"created_by_ref_id\":\"9faf421d-5355-41d9-8731-7f63dc0509ca\"},\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\":{\"extension_type\":\"property-extension\",\"labels\":[\"32-bit\",\"elf\",\"mips\",\"mozi\"],\"description\":\"Malware payload delivery host\",\"score\":60,\"created_by_ref\":\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\"}},\"object_marking_refs\":[\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\"],\"value\":\"123.14.252.72\"}}}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:22:00.163711235Z"}

Any ideas?

from connectors.

I'm no longer getting the errors I had previously, however I am not seeing any data loaded into my tiIndicators in the Defender portal. Do these take a while to get logged in?

from connectors.

And do you know if there are any logs in Defender for Endpoint/Entra that can show me if the upload is successful or not, and if not the issue? I'm seeing no errors in my Sentinel connector at all now, and no indicators uploaded. There is definitely connection as I can see all the successful connection attempts in my sign-in logs.

from connectors.

Hi @blockanz, then I know what you put in the variable ?

• CONNECTOR_LIVE_STREAM_ID=ChangeMe

The two valid cases are :
// General stream

• CONNECTOR_LIVE_STREAM_ID=live

// Stream with filters applied

• CONNECTOR_LIVE_STREAM_ID=(UUID generated by OpenCTI)

If you already have one of these cases, you would need more information on the log side at the connector level, you can replace "error" in "info" for this variable:

• CONNECTOR_LOG_LEVEL=info

from connectors.

I have changed log level and can now see the following:

INFO [CREATE] Processing data {3d4a8c43-87e2-48fc-9134-b975a5e1cecd} | timestamp=2024-04-29T20:55:33.443926Z name=sentinel
INFO [CREATE] ID {3d4a8c43-87e2-48fc-9134-b975a5e1cecd Failed and got }<Response [400]> status code. | timestamp=2024-04-29T20:55:34.083821Z name=sentinel

Any ideas why I am getting a Failed with response [400]? I can see the connection to the API successful when I review the sign-in logs in Entra, application should have the appropriate rights to read/write to DefenderATP graph.

Response 400 suggests bad or malformed request so not sure where that is occurring.

Any help would be greatly appreciated @Megafredo

from connectors.

I made some changes to the application permissions which seems to have resolved some things. Now I am seeing below in the logs:

INFO Starting to listen stream events | timestamp=2024-04-29T22:30:55.949460Z name=sentinel attributes={"live_stream_url":"http://192.168.16.80:8080/stream/1ac36339-a9fd-4a44-b4ad-0bab4a165f08?recover=2024-04-26T01:49:55Z","listen_delete":"false","no_dependencies":"true","with_inferences":"false"}
INFO Initiate work | timestamp=2024-04-29T22:38:08.105670Z name=api attributes={"connector_id":"aaa73d9b-c481-e5e9-d6a7-7acd72df2abb"}
INFO Update action expectations | timestamp=2024-04-29T22:38:08.210204Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:38:08.132Z","expectations":13}
INFO sentinel sending bundle to queue | timestamp=2024-04-29T22:38:08.334139Z name=sentinel
INFO Reporting work update_processed | timestamp=2024-04-29T22:38:08.416719Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:38:08.132Z"}
INFO Initiate work | timestamp=2024-04-29T22:39:09.729092Z name=api attributes={"connector_id":"aaa73d9b-c481-e5e9-d6a7-7acd72df2abb"}
INFO Update action expectations | timestamp=2024-04-29T22:39:09.857278Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:39:09.753Z","expectations":13}
INFO sentinel sending bundle to queue | timestamp=2024-04-29T22:39:09.967434Z name=sentinel
INFO Reporting work update_processed | timestamp=2024-04-29T22:39:10.024125Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:39:09.753Z"}

Unfortunately I am still not seeing indicators reaching Defender, so not sure they are working. Documentation does state that these can take several hours, so I will wait and see if things change.

from connectors.

@blockanz, "when I review the sign-in logs in Entra, application should have the appropriate rights to read/write to DefenderATP graph."

Have you set up the necessary permissions on Sentinel ?

You must have in portal Azure:
Home > Application Registration > OpenCTI (your name) > API Permissions
And prioritize the permissions for "ThreatIndicators.ReadWrite.OwnedBy".

Then you will be able to see the data (indicators) in :
Home > Microsoft Sentinel > OpenCTI (your name) > Threat Intelligence

For more information :
https://learn.microsoft.com/en-us/graph/security-authorization
https://learn.microsoft.com/en-us/azure/sentinel/connect-threat-intelligence-tip

Other interesting link:
https://learn.microsoft.com/en-us/azure/architecture/example-scenario/data/sentinel-threat-intelligence#import-threat-indicators-with-the-platforms-data-connector

from connectors.

Here are my list of application permissions. I am still getting 400 errors.

from connectors.

@blockanz, can you share your docker-compose.yml with me by removing all the important credentials ?

from connectors.