High memory usage iterating over a large Mailbox about exchangelib HOT 5 CLOSED

.all() is greedy and fetches all items in one go. Attachments load their content when accessing attachment.content or attachment.item, which will increase your memory usage while iterating.

If you have attributes that you don't care about, you can optimize your memory usage by fetching only the attributes you need:

for m in account.inbox.all().only('item_id', 'changekey', 'attachments'):
    ...

Finally, res = account.inbox.all() creates an internal cache of the search results so a second iteration over res does not hit the server. If you will never need this cache, you can add .iterator() at the end of your filter, i.e:

for m in account.inbox.all().iterator():
    ...

from exchangelib.

Closed this with a recommendation to use this optimisation and the latest 1.7.5 release:

for m in account.inbox.all().only('item_id', 'changekey', 'attachments').iterator():
    ...
    for attachment in item.attachments:
        ...

If this doesn't help, please provide measurements or other insight into the problem.

from exchangelib.

Does Python crash at 600MB usage? How much raw data is the 5000 messages in total? If your inbox contains large attachments, 600MB may not be unreasonable.

from exchangelib.

638995b clears some elements after XML parsing, which should ease the memory pressure a bit. Can you try if that helps?

A more optimal solution would be to change from a greedy XML parser to an iterating parser, but that would be quite painful.

from exchangelib.

Hi @KarelCoudijzer Did you get to the bottom of this? If you are still having problems, I think #52 would make it possible to create a generator that makes .all() non-greedy.

from exchangelib.