MAJOR: Need to remove boto session caching and replace with credential caching about cloudaux HOT 4 CLOSED

I might be seeing something similar in a different project with botocore. Do you have any links to this issue?

from cloudaux.

Hello @0xdabbad00 : Nothing explicitly linked anywhere. We began noticing some major issues with Security Monkey, where it was effectively crippled.

To resolve, I made a very significant (and very long overdue) refactor to make use of Celery and to segment all watchers into individual processes. The main trick is to have those processes terminate when complete to free up available RAM.

We noticed on our other major project Netflix-Skunkworks/Historical, similar memory leakage. @kevgliss did some major investigative work, and tracked down that somewhere after botocore version 1.7.45(IIRC?), there has been major memory leakage when retaining onto old clients.

Our attempt to permanently resolve this issue is to cache the credentials instead of the boto connections, and to simply create new botocore sessions with the cached credentials each time. We are hopeful that this would help resolve the issue.

from cloudaux.

#59 Fixes this.

from cloudaux.

This is now fixed in #59, version 1.4.7 on PyPI.

from cloudaux.