[ImportFileCsv] Create the connector about connectors HOT 11 CLOSED

@amr-cossi You previously mentioned defining a few templates to set a standard aside. Is that not applicable anymore?

When we bootstrapped our first OpenCTI content from some manual tables, we used the following CSV templates:

• One file to define entities to add to the platform inside one report: report_id, entity_type, name, description, aliases, markings
• One file to define relationships between existing entities inside one report: report_id, from_entitiy_type, from_entitiy_name, to_entitiy_type, to_entitiy_name, relationship_type, [start_time, stop_time, confidence, markings] if different from the report

It was minimalist and helped us on our specific use case but it doesn't seem to be a great example of a first generic CSV importer.

@amr-cossi do you know any tools able to transform csv to STIX?

We don't use any, if the source is a CSV it usually means it's not suited for a raw import in our strategic knowledge database and needs more processing than just a conversion. MISP remains central in the process of technical indicators handling on our side right now.

from connectors.

I was talking to @SyeedHasan a bit and we concluded, that with the current connector setup where the only configuration is possible in the docker compose file, defining different csv templates is very user unfriendly. Giving the user the possibility to modify the config in the OpenCTI UI might be very helpful here.

A temporary workaround until this is resolved, I could extend the import-report connector to simply extract SDOs/SCOs from csv files the same way it works for text files. This way the information at least gets extracted and stored in OpenCTI. However the relationship between the SCOs/SDOs in the same line will be lost (unless simple related-to relationships could be created or something alike). A certain connection persists however, since this export only happens contextually in a report where all extracted entities are referenced to the report anyways.

Would that be something worth implementing or should we wait for a better implementation?

from connectors.

We've started a first version of this connector. The initial goal is to be able to batch create Reports, Entities and Relations based on 3 fixed CSV templates.

from connectors.

@amr-cossi Any update on this connector?

from connectors.

Refer to ImportFileStix and ImportFilePdf ，I have code the ImportFileCsv.
However, when I upload the file, it won't trigger ImportFileCsv.
What should I do?
Best wishes!

from connectors.

Are development efforts required for this? I'd love to chip in some help.

from connectors.

We had developed some scripts to ingest data based on some CSV files for the v3 and they have not been used of migrated for the v4 at all.
I think the first use case that can be developed by anyone is to have a connector which accepts any CSV generated with OpenCTI or with an equivalent format (same columns, some empty).
The more complex use case to be able to ingest a full graph with some name and id resolutions will be very hard to build as a generic connector. It seems safer to have tools to transform some CSV data sources into STIX and use the STIX import connector only.

from connectors.

@amr-cossi You previously mentioned defining a few templates to set a standard aside. Is that not applicable anymore?

Also, we can always scale the connector up from a basic CSV importer to the generic importer you've mentioned.

It seems safer to have tools to transform some CSV data sources into STIX and use the STIX import connector only.

Do you use something in particular to achieve this?

I can't recall for sure but this export-convert-import transformation had an issue. I'll see if I can figure it out again.

from connectors.

It seems safer to have tools to transform some CSV data sources into STIX and use the STIX import connector only.

@amr-cossi do you know any tools able to transform csv to STIX?

from connectors.

One cool-but-not-so-cool way is to import the CSV to AV's OTX and export it as STIX. Works really well. Downside? Private pulses go to OTX as well so if you'd like to keep your intelligence to yourself - that's a bust. @jmau2002

from connectors.

I extended the ImportDocument connector to simply parse CSV files like text file. I'll keep the column specific parsing in mind for later once the the connector base is refactored. Closing this for now

from connectors.