Provide low-throughput alternative for dispatcher cron about binaryalert HOT 3 CLOSED

Thanks for opening this issue!

In the S3 event driven solution, for retroactive analysis could the 'batcher' invoke the analysis lambda directly (skipping the queue)?

from binaryalert.

You are limited by 1,000 concurrently executing Lambda functions in a given region. If you have more than 1,000 files in S3, the batcher will likely be invoking analyzer Lambdas faster than they can be processed, and AWS will throttle you.

This is somewhat mitigated by the fact that each analyzer could still process multiple files, but I don't know if it would be enough.

If you have < 1,000 files, or if you want to add a delay to the batch processing, you could do direct invocation

from binaryalert.

This problem would be alleviated if you could invoke Lambda from SQS, but it's not clear when or if AWS will support this

from binaryalert.