Hi,

my mailbox with ~20k msg's is taking hours to complete.
no idea what changed from 1.3x -> 1.5x but it is way slower.

Show me the numbers in the same conditions but the imapsync release
and I'll take the time to look at and fix this speed issue.

In my opinion perl threads would/could be useful to fasten things up?

Doing things linearly is enough complicated so threads will surely be useful to mess things up.
Anyway the code is too horrible to be threaded.

But it is already possible to parallelize runs on the same mailboxes by choosing
different folders on different parallel runs. Option --include is the key here.

So thread-like behavior, in case it's salvation, is already there.

--include : Sync folders matching this regular expression
--include : or this one, etc.
in case both --include --exclude options are
use, include is done before.
--exclude : Skips folders matching this regular expression
Several folders to avoid:
--exclude 'fold1|fold2|f3' skips fold1, fold2 and f3.
--exclude : or this one, etc.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

from imapsync.

Hi,

it seems that I am wrong.
I was transfering my mailbox from a german provider 1and1 towards gmail and
it seems that gmail imap is incredible slow.

I compared 1.3x with 1.5x directly from 1and1 -> 1and1 and the time did not
really differ

Anyhow I believe that multiple threads (including on imap connection per
thread) could speed up things
I got sth. about 1msg/s and typically for cloud services (like gmail)
scalability is done via horizontally speedup.

but in the end:
+1 for imapsync :-)

Mit freundlich Grüßen / Kind regards

Björn Boschman

from imapsync.

Hi Björn,

I was transfering my mailbox from a german provider 1and1 towards gmail and
it seems that gmail imap is incredible slow.

Gmail has a throttle mechanism.
https://support.google.com/mail/answer/6584

http://imapsync.lamiral.info/FAQ

Q. Synchronizing from XXX to Gmail

R. There are some details to get the special [Gmail] sub-folders
right. Here's an example of migrating an old "Sent" folder to
Gmail's structure:

imapsync --host1 mail.oldhost.com
--user1 [email protected]
--password1 password
--host2 imap.gmail.com
--user2 [email protected]
--password2 password
--ssl2
--exitwhenover 500000000
--maxsize 25000000
--expunge1
--addheader
--exclude "[Gmail]$"
--regextrans2 "s/[ ]+//g"
--regextrans2 "s/[^]//g"
--regextrans2 "s/['"\]/_/g"
--regextrans2 "s,^Sent$,[Gmail]/Sent Mail,"

Explanations:

--ssl2 is mandatory since Gmail only supports imap ssl connections.

--exitwhenover 500000000 option is here to avoid locking when
transfers exceed maximum limit.
See http://support.google.com/a/bin/answer.py?hl=en&answer=1071518
--exitwhenover is not mandatory in the sense you may be able to
use an upper value than 500 MB without disconnections; I don't
know the hard value, it seems to vary, so just have some tries
and report me what you discover in case you detect something.

--maxsize 25000000 is mandatory since Gmail limits messages size
up to 25 MB. This value increases over time, it was 10 MB some
years ago so you can try higher values. The Gmail page about
this limit is https://support.google.com/mail/answer/6584

--expunge1 is optional. It deletes messages marked \Deleted on host1.
Imapsync syncs messages with all their flags, Gmail takes the messages
marked \Deleted but deletes or moves them just after.
Option --expunge1 really removes messages marked \Deleted on host1
so they are not synced at all.

The --addheader option is there because "Sent" folder messages
sometimes lack the "Message-Id:" and "Received:" headers needed
by imapsync to identify messages (only when --useuid is not used).
So option --addheader adds a "Message-Id" header consisting of
the imap UID of the message on the host1 folder, like
"Message-Id: 12345@imapsync".

--exclude "[Gmail]$" is there to avoid a small examine/select error:
"Could not examine: 43 NO [NONEXISTENT] Unknown Mailbox: [Gmail](now in authenticated state) (Failure)".

--regextrans2 "s/[ ]+/_/g" is there to convert blank characters not
accepted by gmail to character _ underscore. In fact only leading
and trailing blank characters are problems with gmail, and
also successive blanks ending with the IMAP error
"NO [CANNOT] Folder contains excess whitespace (Failure)"
If you want to change only leading and trailing blank characters
then use instead:
--regextrans2 "s,(/|^) +,$1,g" --regextrans2 "s, +(/|$),$1,g"

On Windows, in the previous example containing $1 you have to
replace the two $1 by $1 (remove the \ before $1).

--regextrans2 "s/[^]/_/g" is mandatory. It converts, since
not accepted by gmail, character ^ to character _ underscore.

--regextrans2 "s/['"\]/_/g" is optional. It converts
characters ' or " or \ to character _ underscore.

--regextrans2 "s,^Sent$,[Gmail]/Sent Mail," is to transform the
folder name "Sent" and adapt it to Gmail "Sent Mail" folder.
If you're using a different language in Gmail you might adapt
this example with the folder name translated, an example in French:

imapsync ...
--regextrans2 "s,^Messages envoy&AOk-s$,[Gmail]/Messages envoy&AOk-s," \

You can add --folder "INBOX.Sent" in the example in case
you want to sync only the "Sent" folder.

You can select folders exported to imap within the gmail preferences,
unselect some "System labels", depending on your needs.

The "All Mail" archive pseudo-folder should be updated automatically.

I compared 1.3x with 1.5x directly from 1and1 -> 1and1 and the time did not
really differ

Good.

Anyhow I believe that multiple threads (including on imap connection per
thread) could speed up things

It's true.
Anyway imapsync is far from being ready for threading.
The code is horrible. I know what I mean, I wrote that mess.

I got sth. about 1msg/s and typically for cloud services (like gmail)
scalability is done via horizontally speedup.

It's done easily by several imapsync instances and --include

Thanks for your honesty!

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

from imapsync.