Comments (13)
Hi @blockanz, it seems that there is another environment variable missing in your yml, given the error I would say :
- RESOURCE_URL=https://graph.microsoft.com
- REQUEST_URL=/beta/security/tiIndicators
Here is the link to docker-compose with all the environment variables for sentinel, can you compare it with the one you have ?
I hope this solves your problem.
from connectors.
Grrr. I somehow mistyped things and had the INCIDENT_URL with the REQUEST_URL value. Thanks you. I'll test this with the proper values and advise.
from connectors.
@The-Stuke I saw that you created the connector. Do you know what's happening?
Otherwise @Megafredo or @helene-nguyen could you have a look when you have time? This is a connector being under the community supervision FYI
from connectors.
Hi @blockanz, this error occurs when the environment variable "EXPIRE_TIME" is either missing or empty, can you check this variable in your .yml ?
By default in the README:
EXPIRE_TIME=30
from connectors.
I made the change and added EXPIRE_TIME=30.
Now I get the following errors:
{"log":"{"timestamp": "2024-04-25T01:21:59.424348Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Failed processing data {can only concatenate str (not \"NoneType\") to str}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.42478946Z"}
{"log":"{"timestamp": "2024-04-25T01:21:59.425073Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Message data {{\"version\":\"4\",\"type\":\"create\",\"scope\":\"external\",\"message\":\"creates a IPv4-Addr 123.14.18.239
\",\"origin\":{\"socket\":\"query\",\"ip\":\"::ffff:192.168.48.1\",\"user_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"group_ids\":[\"576aa993-0257-46cf-844d-8d5a44128257\"],\"organization_ids\":[],\"user_metadata\":{},\"applicant_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"call_retry_number\":\"1\"},\"data\":{\"id\":\"ipv4-addr--21075343-2f26-5461-9993-263f210858ff\",\"spec_version\":\"2.1\",\"type\":\"ipv4-addr\",\"extensions\":{\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\":{\"extension_type\":\"property-extension\",\"id\":\"94ce7581-6907-41e3-a065-0c9a27bfba74\",\"type\":\"IPv4-Addr\",\"created_at\":\"2024-04-25T01:21:58.817Z\",\"updated_at\":\"2024-04-25T01:21:58.817Z\",\"is_inferred\":false,\"creator_ids\":[\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\"],\"labels_ids\":[\"c13f46fe-addf-4d20-9907-dbc599753220\",\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\"],\"created_by_ref_id\":\"9faf421d-5355-41d9-8731-7f63dc0509ca\"},\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\":{\"extension_type\":\"property-extension\",\"labels\":[\"elf\",\"mozi\"],\"description\":\"Malware payload delivery host\",\"score\":60,\"created_by_ref\":\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\"}},\"object_marking_refs\":[\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\"],\"value\":\"123.14.18.239\"}}}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.42533194Z"}
{"log":"{"timestamp": "2024-04-25T01:21:59.879506Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Failed processing data {can only concatenate str (not \"NoneType\") to str}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.880151227Z"}
{"log":"{"timestamp": "2024-04-25T01:21:59.881081Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Message data {{\"version\":\"4\",\"type\":\"create\",\"scope\":\"external\",\"message\":\"creates a IPv4-Addr 123.14.251.202
\",\"origin\":{\"socket\":\"query\",\"ip\":\"::ffff:192.168.48.1\",\"user_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"group_ids\":[\"576aa993-0257-46cf-844d-8d5a44128257\"],\"organization_ids\":[],\"user_metadata\":{},\"applicant_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"call_retry_number\":\"1\"},\"data\":{\"id\":\"ipv4-addr--305c4cae-d829-5ee5-a850-c8fe145146a1\",\"spec_version\":\"2.1\",\"type\":\"ipv4-addr\",\"extensions\":{\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\":{\"extension_type\":\"property-extension\",\"id\":\"cf824fcc-d364-499c-8311-f5e9e3e84126\",\"type\":\"IPv4-Addr\",\"created_at\":\"2024-04-25T01:21:59.351Z\",\"updated_at\":\"2024-04-25T01:21:59.351Z\",\"is_inferred\":false,\"creator_ids\":[\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\"],\"labels_ids\":[\"c13f46fe-addf-4d20-9907-dbc599753220\",\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\"],\"created_by_ref_id\":\"9faf421d-5355-41d9-8731-7f63dc0509ca\"},\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\":{\"extension_type\":\"property-extension\",\"labels\":[\"elf\",\"mozi\"],\"description\":\"Malware payload delivery host\",\"score\":60,\"created_by_ref\":\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\"}},\"object_marking_refs\":[\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\"],\"value\":\"123.14.251.202\"}}}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:21:59.881292975Z"}
{"log":"{"timestamp": "2024-04-25T01:22:00.162560Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Failed processing data {can only concatenate str (not \"NoneType\") to str}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:22:00.162989455Z"}
{"log":"{"timestamp": "2024-04-25T01:22:00.163477Z", "level": "ERROR", "name": "sentinel", "message": "[ERROR] Message data {{\"version\":\"4\",\"type\":\"create\",\"scope\":\"external\",\"message\":\"creates a IPv4-Addr 123.14.252.72
\",\"origin\":{\"socket\":\"query\",\"ip\":\"::ffff:192.168.48.1\",\"user_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"group_ids\":[\"576aa993-0257-46cf-844d-8d5a44128257\"],\"organization_ids\":[],\"user_metadata\":{},\"applicant_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"call_retry_number\":\"1\"},\"data\":{\"id\":\"ipv4-addr--d11fbddd-56a6-5f3a-ac93-0456a333fcd6\",\"spec_version\":\"2.1\",\"type\":\"ipv4-addr\",\"extensions\":{\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\":{\"extension_type\":\"property-extension\",\"id\":\"04dd40dc-6b01-46e4-9c37-bc511669cd10\",\"type\":\"IPv4-Addr\",\"created_at\":\"2024-04-25T01:21:59.435Z\",\"updated_at\":\"2024-04-25T01:21:59.435Z\",\"is_inferred\":false,\"creator_ids\":[\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\"],\"labels_ids\":[\"4491d7c7-5744-408e-aa4b-837dd2dd172d\",\"c13f46fe-addf-4d20-9907-dbc599753220\",\"42c9846a-d05b-4bf4-9956-236dfdae90e6\",\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\"],\"created_by_ref_id\":\"9faf421d-5355-41d9-8731-7f63dc0509ca\"},\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\":{\"extension_type\":\"property-extension\",\"labels\":[\"32-bit\",\"elf\",\"mips\",\"mozi\"],\"description\":\"Malware payload delivery host\",\"score\":60,\"created_by_ref\":\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\"}},\"object_marking_refs\":[\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\"],\"value\":\"123.14.252.72\"}}}", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 458, in _process_message\n self._create_observable(data)\n File \"/opt/opencti-connector-sentinel/sentinel.py\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~\nTypeError: can only concatenate str (not \"NoneType\") to str"}\n","stream":"stderr","time":"2024-04-25T01:22:00.163711235Z"}
Any ideas?
from connectors.
I'm no longer getting the errors I had previously, however I am not seeing any data loaded into my tiIndicators in the Defender portal. Do these take a while to get logged in?
from connectors.
And do you know if there are any logs in Defender for Endpoint/Entra that can show me if the upload is successful or not, and if not the issue? I'm seeing no errors in my Sentinel connector at all now, and no indicators uploaded. There is definitely connection as I can see all the successful connection attempts in my sign-in logs.
from connectors.
Hi @blockanz, then I know what you put in the variable ?
- CONNECTOR_LIVE_STREAM_ID=ChangeMe
The two valid cases are :
// General stream
- CONNECTOR_LIVE_STREAM_ID=live
// Stream with filters applied
- CONNECTOR_LIVE_STREAM_ID=(UUID generated by OpenCTI)
If you already have one of these cases, you would need more information on the log side at the connector level, you can replace "error" in "info" for this variable:
- CONNECTOR_LOG_LEVEL=info
from connectors.
I have changed log level and can now see the following:
INFO [CREATE] Processing data {3d4a8c43-87e2-48fc-9134-b975a5e1cecd} | timestamp=2024-04-29T20:55:33.443926Z name=sentinel
INFO [CREATE] ID {3d4a8c43-87e2-48fc-9134-b975a5e1cecd Failed and got }<Response [400]> status code. | timestamp=2024-04-29T20:55:34.083821Z name=sentinel
Any ideas why I am getting a Failed with response [400]? I can see the connection to the API successful when I review the sign-in logs in Entra, application should have the appropriate rights to read/write to DefenderATP graph.
Response 400 suggests bad or malformed request so not sure where that is occurring.
Any help would be greatly appreciated @Megafredo
from connectors.
I made some changes to the application permissions which seems to have resolved some things. Now I am seeing below in the logs:
INFO Starting to listen stream events | timestamp=2024-04-29T22:30:55.949460Z name=sentinel attributes={"live_stream_url":"http://192.168.16.80:8080/stream/1ac36339-a9fd-4a44-b4ad-0bab4a165f08?recover=2024-04-26T01:49:55Z","listen_delete":"false","no_dependencies":"true","with_inferences":"false"}
INFO Initiate work | timestamp=2024-04-29T22:38:08.105670Z name=api attributes={"connector_id":"aaa73d9b-c481-e5e9-d6a7-7acd72df2abb"}
INFO Update action expectations | timestamp=2024-04-29T22:38:08.210204Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:38:08.132Z","expectations":13}
INFO sentinel sending bundle to queue | timestamp=2024-04-29T22:38:08.334139Z name=sentinel
INFO Reporting work update_processed | timestamp=2024-04-29T22:38:08.416719Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:38:08.132Z"}
INFO Initiate work | timestamp=2024-04-29T22:39:09.729092Z name=api attributes={"connector_id":"aaa73d9b-c481-e5e9-d6a7-7acd72df2abb"}
INFO Update action expectations | timestamp=2024-04-29T22:39:09.857278Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:39:09.753Z","expectations":13}
INFO sentinel sending bundle to queue | timestamp=2024-04-29T22:39:09.967434Z name=sentinel
INFO Reporting work update_processed | timestamp=2024-04-29T22:39:10.024125Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:39:09.753Z"}
Unfortunately I am still not seeing indicators reaching Defender, so not sure they are working. Documentation does state that these can take several hours, so I will wait and see if things change.
from connectors.
@blockanz, "when I review the sign-in logs in Entra, application should have the appropriate rights to read/write to DefenderATP graph."
Have you set up the necessary permissions on Sentinel ?
You must have in portal Azure:
Home > Application Registration > OpenCTI (your name) > API Permissions
And prioritize the permissions for "ThreatIndicators.ReadWrite.OwnedBy".
Then you will be able to see the data (indicators) in :
Home > Microsoft Sentinel > OpenCTI (your name) > Threat Intelligence
For more information :
https://learn.microsoft.com/en-us/graph/security-authorization
https://learn.microsoft.com/en-us/azure/sentinel/connect-threat-intelligence-tip
Other interesting link:
https://learn.microsoft.com/en-us/azure/architecture/example-scenario/data/sentinel-threat-intelligence#import-threat-indicators-with-the-platforms-data-connector
from connectors.
Here are my list of application permissions. I am still getting 400 errors.
from connectors.
@blockanz, can you share your docker-compose.yml with me by removing all the important credentials ?
from connectors.
Related Issues (20)
- Malware Bazar Connector Error HOT 2
- [Export CSV] Relationships export with filter exports more than required HOT 2
- Error when trying to export without order by
- Using a Python remote debugger for docker-compose local development HOT 1
- Qradar-Connector Deprecated API HOT 3
- [import/export] Error when importing text files or PDFs
- [Malpedia] ERROR: User effective max confidence level is insufficient to update this element HOT 6
- Stream Connector goes to "Inactive" status, does not "StreamAlive" HOT 2
- [Export CSV] Export both FROM NAME and TO NAME
- External reference not (re)created when using bundle and OpenCTIStix2.put_attribute_in_extension HOT 1
- Add new sources to Recorded Future external-import HOT 1
- Add additional debugging statements to the Recorded Future connectors
- [Import] Validating a workbench within a report can import a random file into it. HOT 5
- MISP AUTH Key variable appears to be missing for the misp-feed connector HOT 2
- Virus Total connector fails to enrich observables with an empty score HOT 1
- connector-diode-import - Container Issue - python3: can't open file '//diode-import.py': HOT 5
- [Intel471] List of bug and improvement HOT 6
- MISP integration is not working HOT 4
- [Mandiant] Remove empty report imported with Indicator
- New Connectors not showing up - Please help :)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connectors.