Make TIProvider run async and report progress with tqdm about msticpy HOT 1 CLOSED

Rough plan:

TI Providers are of 2 types:

• those that support one observable at a time
• those that support queries with multiple observables (e.g. Sentinel provider)

The architecture is currently

• TILookup - (with main funcs lookup_ioc and lookup_iocs) is a facade/proxy class that handles dispatching calls to each configured provider
  • it simple iterates through the providers in a for loop and combines the results to be returned in a DF
• ti_providers are derived from ti_provider_base. The lookup_iocs implemented in ti_provider_base simply iterates (via a generator) through the supplied iterable/DF of observables, calling the lookup_ioc method implemented for the specific provider.

Constraints:

• we probably don't want to default to hammering TI providers with a big set of async requests since we'll probably quickly hit request limits.
• some users may have enterprise licenses and want the requests for some providers to be async - so maybe we need handle async requests differently depending on user configuration.
• can we handle per-provider request limits? What would this look like? Process N reqs per minute then sleep until minute expires? (N is req limit/min)

Proposed solution:

Phase 1

• in TILookup create an async task list for each provider
• For multiple requests, split the list into some multiple of batches (for feedback via TQDM) - e.g. 10
• TQDM look tracks # of batches (not # of providers)
• Have synchronous for loop handle request chunks
• Dispatch to async tasks for each configured provider.

Phase 2

• Allow individual providers to be configured for async
• Re-implement async version of ti_provider_base.lookup_iocs that can dispatch batch of requests asynchronously
• When called with async=True (config) send via async
• Otherwise just look synchronously.

from msticpy.