Frightening amount of pending analysis about peekabooav HOT 9 CLOSED

Hi @MigliS,
there is only a few reasons why Cuckoo builds up a pending analysis queue.
A file submited to Cuckoo is only an entry in its database. That is then displayed by WebUI.

1. It could be the case that Cuckoo is not running at all,
2. max parallel analysis count is reached or
3. all available analysis VMs are in use.

Check if there are:

1. successfull reports for the time period,
2. if your VMs are reachable over host only network,
3. if Cuckoo is running properly (and Cuckoo processing units)

Best
Felix

PS:
Please share your rulset changes and reasoning

from peekabooav.

Hi Felix

Excuse me for my late reply. I still cannot tell where this amount is coming from, but now i changed the Cuckoo submit command from /usr/bin/cuckoo submit to /usr/bin/cuckoo submit --unique which will only submit files once to Cuckoo. After this configuration all looked pretty good.

Changes I made in the PeekabooAV ruleset peekaboo/peekaboo/ruleset/rules.py are for example deactivating MIME-types like image/jpg, image/png and PowerPoint files.
Normally our incoming mails have MIME-types with signatures etc. which I also put on the whitelist.

Would you recomment to put pdf on the whitelist or leave on the greylist? That's one point I'm really not sure about.

Regards
Michael

from peekabooav.

Every parts of the mails goes into the database like I can see.

Log says :

Feb 02 16:44:58 hostname python[20847]: 2018-02-02 16:44:58,395 - peekaboo.sample - (Thread-34) - ERROR - No section: 'attachment'
Feb 02 16:44:58 hostname python[20847]: Traceback (most recent call last):
Feb 02 16:44:58 hostname python[20847]: File "build/bdist.linux-x86_64/egg/peekaboo/sample.py", line 315, in mimetypes
Feb 02 16:44:58 hostname python[20847]: declared_mt = self.__meta_info.get_mime_type()
Feb 02 16:44:58 hostname python[20847]: File "build/bdist.linux-x86_64/egg/peekaboo/toolbox/sampletools.py", line 66, in get_mime_type
Feb 02 16:44:58 hostname python[20847]: return self.meta_info.get('attachment', 'type_declared')
Feb 02 16:44:58 hostname python[20847]: File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
Feb 02 16:44:58 hostname python[20847]: raise NoSectionError(section)
Feb 02 16:44:58 hostname python[20847]: NoSectionError: No section: 'attachment'
Feb 02 16:44:58 hostname python[20847]: 2018-02-02 16:44:58,407 - peekaboo.ruleset.engine - (Thread-34) - INFO - Rule 'file_type_on_whitelist' processed for <Sample(filename='p007', known='no', meta_info_loaded='yes', job_id='-1', result='Result.inProgress', sha256sum='f4e43727f27d5b501ccb326639cc27600dd778edd274e6c55b484ac666b5d2fb')>
Feb 02 16:44:58 hostname python[20847]: 2018-02-02 16:44:58,413 - peekaboo.sample - (Thread-34) - ERROR - No section: 'attachment'
Feb 02 16:44:58 hostname python[20847]: Traceback (most recent call last):
Feb 02 16:44:58 hostname python[20847]: File "build/bdist.linux-x86_64/egg/peekaboo/sample.py", line 315, in mimetypes
Feb 02 16:44:58 hostname python[20847]: declared_mt = self.__meta_info.get_mime_type()
Feb 02 16:44:58 hostname python[20847]: File "build/bdist.linux-x86_64/egg/peekaboo/toolbox/sampletools.py", line 66, in get_mime_type
Feb 02 16:44:58 hostname python[20847]: return self.meta_info.get('attachment', 'type_declared')
Feb 02 16:44:58 hostname python[20847]: File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
Feb 02 16:44:58 hostname python[20847]: raise NoSectionError(section)
Feb 02 16:44:58 hostname python[20847]: NoSectionError: No section: 'attachment'
Feb 02 16:44:58 hostname python[20847]: 2018-02-02 16:44:58,421 - peekaboo.sample - (Thread-34) - ERROR - No section: 'attachment'
Feb 02 16:44:58 hostname python[20847]: Traceback (most recent call last):
Feb 02 16:44:58 hostname python[20847]: File "build/bdist.linux-x86_64/egg/peekaboo/sample.py", line 315, in mimetypes
Feb 02 16:44:58 hostname python[20847]: declared_mt = self.__meta_info.get_mime_type()
Feb 02 16:44:58 hostname python[20847]: File "build/bdist.linux-x86_64/egg/peekaboo/toolbox/sampletools.py", line 66, in get_mime_type
Feb 02 16:44:58 hostname python[20847]: return self.meta_info.get('attachment', 'type_declared')
Feb 02 16:44:58 hostname python[20847]: File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
Feb 02 16:44:58 hostname python[20847]: raise NoSectionError(section)
Feb 02 16:44:58 hostname python[20847]: NoSectionError: No section: 'attachment'

from peekabooav.

Hi Michael,

to your first question. It's totally up to you, if you're not afraid of PDF files you don't need to scan them. If you have static checks for JS in open action or alike ... you might decide to not scan PDF files. Or only scan those with JS or forms ... you need to evaluate risk and available resources.

The log you included shows indications that the amavis dump_info can't be parsed properly. My assumption is that the files is created but its content is in someway malformed maybe with an error message or similar.

Please let me know your findings.

Best
Felix

from peekabooav.

Hi Felix

Thank you for helping with my concerns :) I'm going to make a few performance tests and will then decide, if we want to scan pdf files.

Second, until now I couldn't find an error in amavis or postfix log.
My last thought was a wrong rules file but it was correctly.

Regards
Michael

from peekabooav.

@MigliS any news on this?
I will close this issue now, feel free to reopen

from peekabooav.

Hi @Jack28
As mentioned /usr/bin/cuckoo submit --unique solves the problem a bit.
In my bulletin that I've sent you, I could figure out MySQL (MariaDB) has a standard max_connections setting which was set to 100. With our incoming traffic this has been to minor.

Regards
Michael

from peekabooav.

OK. Thanks for your solution.
Right now I can't think of a reason how this should address the issue but I'm glad it did.
So far we can't reproduce this error. It might only occure on your setup and useage.
Sorry that I can't give you any more on this.
I will keep it in mind and comment as soon as I come across it again.

from peekabooav.

Hi Felix
I just remembered you told, we should only try to analyze pdf documents which contain forms or js.

Hi Michael,
to your first question. It's totally up to you, if you're not afraid of PDF files you don't need to scan them. If you have static checks for JS in open action or alike ... you might decide to not scan PDF files. Or only scan those with JS or forms ... you need to evaluate risk and available resources.
The log you included shows indications that the amavis dump_info can't be parsed properly. My assumption is that the files is created but its content is in someway malformed maybe with an error message or similar.
Please let me know your findings.
Best
Felix

Do you even know how it is possible to get this kind of "Content-Type". As far as I know there's only the mime type application/pdf, so I can't imagine how we're able to analyze only those files.
What are your concerns about that?

regards
Michael

from peekabooav.