offlineimap / offlineimap3 Goto Github PK

View Code? Open in Web Editor NEW

391.0 23.0 57.0 8.75 MB

Read/sync your IMAP mailboxes (python3)

License: Other

Makefile 0.41% Python 97.16% Shell 2.24% Dockerfile 0.19%

offlineimap3's Introduction

Upstream status (master branch):

Upstream status (next branch):

Links:

Official github code repository
- for Python 2: offlineimap
- for Python 3: offlineimap3
Website: website
Wiki: wiki
Blog: blog

OfflineIMAP

"Get the emails where you need them."

Description

OfflineIMAP is software that downloads your email mailbox(es) as local Maildirs. OfflineIMAP will synchronize both sides via IMAP.

Why should I use OfflineIMAP?

IMAP's main downside is that you have to trust your email provider to not lose your email. While certainly unlikely, it's not impossible. With OfflineIMAP, you can download your Mailboxes and make you own backups of your Maildir.

This allows reading your email offline without the need for your mail reader (MUA) to support IMAP operations. Need an attachment from a message without internet connection? No problem, the message is still there.

Project status and future

OfflineIMAP, using Python 3, is based on OfflineIMAP for Python 2. Currently we are updating the source code. These changes should not affect the user (documentation, configuration files,... are the same) but some links or packages could refer to the Python 2 version. In that case, please open an issue.

License

GNU General Public License v2.

Downloads

You should first check if your distribution already packages OfflineIMAP for you. Downloads releases as tarball or zipball.

If you are running Linux, you can install offlineimap with:

Debian and Ubuntu apt install offlineimap3
openSUSE zypper install offlineimap
Fedora dnf install offlineimap
Arch Linux: pacman -S offlineimap, or through AUR package offlineimap3-git
Docker image: offlineimap/offlineimap:latest (note: image not published yet, just an example)

Feedbacks and contributions

The user discussions, development, announcements and all the exciting stuff take place on the mailing list. While not mandatory to send emails, you can subscribe here.

Bugs, issues and contributions can be requested to both the mailing list or the official Github project. Provide the following information:

system/distribution (with version)
offlineimap version (offlineimap -V)
Python version
server name or domain
CLI options
Configuration file (offlineimaprc)
pythonfile (if any)
Logs, error
Steps to reproduce the error

The community

OfflineIMAP's main site is the project page at Github.
There is the OfflineIMAP community's website.
And finally, the wiki.

Requirements & dependencies

Python v3+
rfc6555 (required)
imaplib2 >= 3.5
keyring
gssapi (optional), for Kerberos authentication
portalocker (optional), if you need to run offlineimap in Cygwin for Windows

Documentation

All current and updated documentation is on the community's website.

Read documentation locally

You might want to read the documentation locally. Get the sources of the website. For the other documentation, run the appropriate make target:

$ ./scripts/get-repository.sh website
$ cd docs
$ make html  # Requires rst2html
$ make man   # Requires a2x (http://asciidoc.org)
$ make api   # Requires sphinx

offlineimap3's People

Contributors

Stargazers

Watchers

offlineimap3's Issues

Fails to establish IMAP connection across preauth tunnel

General informations

system/distribution (with version): Debian unstable
offlineimap version (offlineimap -V): 7.3.0 (Debian 0.0~git20201025.c850e74+dfsg-1)
Python version: v3.8.6

I've been using OfflineIMAP with a SSH preauth tunnel for many
years. Unfortunately, offlineimap3 doesn't work. It seems that it
never sees the IMAP banner:

% offlineimap -o -d ALL
OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.8.6, OpenSSL 1.1.1h  22 Sep 2020
Now debugging for imap: IMAP protocol debugging
Now debugging for maildir: Maildir repository debugging
Now debugging for thread: Threading debugging
Now debugging for : Other offlineimap related sync messages
[thread]: Register new thread 'Account sync madduck.net' (account 'madduck.net')
[imap]: Using authentication mechanisms ['GSSAPI', 'XOAUTH2', 'CRAM-MD5', 'PLAIN', 'LOGIN']
[maildir]: MaildirRepository initialized, sep is '/'
*** Processing account madduck.net
Establishing connection to tunnel:.offlineimap/preauthtunnel.sh madduck-net.imap.madduck.net (madduck.net)
OpenSSH_8.4p1 Debian-2, OpenSSL 1.1.1h  22 Sep 2020
debug1: Reading configuration data .offlineimap/ssh_config
debug1: .offlineimap/ssh_config line 1: Applying options for madduck-net.imap.madduck.net
debug3: expanded UserKnownHostsFile '~/.offlineimap/known_hosts' -> '/home/madduck/.offlineimap/known_hosts'
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/madduck/.var/offlineimap/ssh_ctl_sock" does not exist
debug2: resolving "madduck-net.imap.madduck.net" port 22
debug2: ssh_connect_direct
debug1: Connecting to madduck-net.imap.madduck.net [2001:a60:902f::bcae:fda6] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 59710 ms remain after connect
debug1: identity file .offlineimap/madduck-net.imap.madduck.net.ssh-seckey type 3
debug1: identity file .offlineimap/madduck-net.imap.madduck.net.ssh-seckey-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to madduck-net.imap.madduck.net:22 as 'madduck'
debug3: hostkeys_foreach: reading file "/home/madduck/.offlineimap/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/madduck/.offlineimap/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from madduck-net.imap.madduck.net
debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],ssh-ed25519,[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: [email protected],zlib,none
debug2: compression stoc: [email protected],zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: [email protected]
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: [email protected]
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:lOTlgZ5YuFXt7SqKQ4WJWnY2mtDtNr7So3tJtku5gfs
debug3: hostkeys_foreach: reading file "/home/madduck/.offlineimap/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/madduck/.offlineimap/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from madduck-net.imap.madduck.net
debug3: hostkeys_foreach: reading file "/home/madduck/.offlineimap/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/madduck/.offlineimap/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 2001:a60:902f::bcae:fda6
debug1: Host 'madduck-net.imap.madduck.net' is known and matches the ECDSA host key.
debug1: Found key in /home/madduck/.offlineimap/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: .offlineimap/madduck-net.imap.madduck.net.ssh-seckey ED25519 SHA256:XbZGAVwGDMSBZ7Kw0Ldvc1rs1DQ7VPj/oyZU1EoN5Ms explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: .offlineimap/madduck-net.imap.madduck.net.ssh-seckey ED25519 SHA256:XbZGAVwGDMSBZ7Kw0Ldvc1rs1DQ7VPj/oyZU1EoN5Ms explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: .offlineimap/madduck-net.imap.madduck.net.ssh-seckey ED25519 SHA256:XbZGAVwGDMSBZ7Kw0Ldvc1rs1DQ7VPj/oyZU1EoN5Ms explicit
debug3: sign_and_send_pubkey: ED25519 SHA256:XbZGAVwGDMSBZ7Kw0Ldvc1rs1DQ7VPj/oyZU1EoN5Ms
debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:XbZGAVwGDMSBZ7Kw0Ldvc1rs1DQ7VPj/oyZU1EoN5Ms
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to madduck-net.imap.madduck.net ([2001:a60:902f::bcae:fda6]:22).
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /home/madduck/.var/offlineimap/ssh_ctl_sock.InQlgz8c2mIEMoXe
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new [/home/madduck/.var/offlineimap/ssh_ctl_sock]
debug3: muxserver_listen: mux listener channel 0 fd 4
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 1: new [client-session]
debug3: ssh_session2_open: channel_new: 1
debug2: channel 1: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: id
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/madduck/.ssh/authorized_keys:23: key options: command user-rc
debug3: receive packet: type 4
debug1: Remote: /home/madduck/.ssh/authorized_keys:23: key options: command user-rc
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 1: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x08
debug2: client_session2_setup: id 1
debug2: channel 1: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 1: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug2: channel 1: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 1
debug2: shell request accepted on channel 1
debug3: send packet: type 80
debug3: receive packet: type 82
[…]
debug3: send packet: type 80
debug3: receive packet: type 82

Those two lines then repeat endlessly every few seconds (they are
keepalives). There is nothing in the server logs, but if I run
sshd in debug mode, I can see that it actually runs the imapd
process:

Starting session: forced-command (key-option) 'MAIL=$HOME/.maildir /usr/lib/dovecot/imap 2>/dev/null' for madduck from 2404:130:0:1000:ce46:cc29:f96:533d port 35656 id 0

And in fact, if I invoke the preauth tunnel wrapper (which is just a
wrapper around ssh), I see the IMAP greeting just fine:

lotus:~% SSH_AUTH_SOCK= .offlineimap/preauthtunnel.sh madduck-net.imap.madduck.net
* PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in as madduck

What is interesting is that when I invoke the preauth tunnel from
OfflineIMAP, I get this debug output from SSH:

[…]
debug1: Entering interactive session.
debug1: pledge: id
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/madduck/.ssh/authorized_keys:23: key options: command user-rc
debug3: receive packet: type 4
debug1: Remote: /home/madduck/.ssh/authorized_keys:23: key options: command user-rc
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 1: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x08
debug2: client_session2_setup: id 1
debug2: channel 1: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 1: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug2: channel 1: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 1
debug2: shell request accepted on channel 1
debug2: channel 1: rcvd ext data 338
Environment:
  USER=madduck
  LOGNAME=madduck
  HOME=/home/madduck
  PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
  MAIL=/var/mail/madduck
  SHELL=/bin/zsh
  LANG=en_NZ.UTF-8
  LANGUAGE=en_NZ:en
  SSH_CLIENT=2404:130:0:1000:ce46:cc29:f96:533d 35656 22
  SSH_CONNECTION=2404:130:0:1000:ce46:cc29:f96:533d 35656 2001:a60:902f::bcae:fda6 22
debug2: channel 1: written 338 to efd 7
debug2: channel 1: rcvd ext data 38
Running /bin/zsh -c '/bin/sh .ssh/rc'
debug2: channel 1: written 38 to efd 7

whereas if I run preauthtunnel directly, I get:

debug1: Entering interactive session.
debug1: pledge: id
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/madduck/.ssh/authorized_keys:23: key options: command user-rc
debug3: receive packet: type 4
debug1: Remote: /home/madduck/.ssh/authorized_keys:23: key options: command user-rc
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 1: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IPV6_TCLASS 0x08
debug2: client_session2_setup: id 1
debug2: channel 1: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 1: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug2: channel 1: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 1
debug2: shell request accepted on channel 1
* PREAUTH [CAPABILITY IMAP4rev1 […]

If I switch back to the Python2 version (7.3.3, Debian 7.3.3+dfsg1-1), it just works again, same config.

imaplib2 issue in docker container

General informations

system/distribution (with version): Python 3-alpine docker image
offlineimap version (offlineimap -V): 7.3.3
Python version: 3.10.0

Logs, error

File "/usr/src/offlineimap3/offlineimap/init.py", line 29, in

import offlineimap.virtual_imaplib2 as imaplib

File "/usr/src/offlineimap3/offlineimap/virtual_imaplib2.py", line 36, in

if (int(imaplib.__release__) < _SUPPORTED_RELEASE or

AttributeError: module 'imaplib2' has no attribute 'release'

Steps to reproduce the error

I am trying to get offlineimap3 running in a docker container. But when trying to run the container I always get the error message :
" if (int(imaplib.release) < _SUPPORTED_RELEASE or AttributeError: module 'imaplib2' has no attribute 'release'"
I am using imaplib2 version > 3.5 as listed in the prerequisites.
Is this a known bug or am I doing something wrong?

Where to get imaplib2 dependency from?

Hey there, question:

Where do I get the correct version of imaplib2 from?

I see that there is a version solely used for oflineimap2:
https://github.com/offlineimap/offlineimap3/blob/master/Changelog.md#notes

What version do you use for offlineimap3?

Thanks for the effort here, btw!

Make offlineimap accept system CA certs by default

Having seen the report in https://alioth-lists.debian.net/pipermail/offlineimap-project/2021-October/021276.html
and checking the default offlineimap.conf at

offlineimap3/offlineimap.conf

Line 757 in e64c254

# Special value OS-DEFAULT makes Offlineimap to automatically

I wonder if the special value OS-DEFAULT
should not be the default value, so that offlineimap trusts those certificates by default that are also trusted by the operating system.

It is a sensible default and would make configuration much easier for everyone who uses for instance letsencrypt certs.

version bump for offlineimap3

Hi,
I am planning to package offlineimap3 for Debian (https://bugs.debian.org/970623) and we already have offlineimap3 v7.3.3 in Debian. offlineimap3 has a version of v7.3.0 so that will be a problem, if I update offlineimap package with this. Is there any plan to bump the version to something like v7.4.0-rc1 ?

--
Regards
Sudip

GSSAPI/Kerberos authentication broken

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): 0.0~git20210105.00d395b+dfsg-2
Debian bug: https://bugs.debian.org/810119 and https://bugs.debian.org/969298

Note:
Issue was seen with offlineimap and is also seen with offlineimap3, not sure why OfflineIMAP/offlineimap#332 was closed.

Logs, error

ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'honk.sigxcpu.org'
  sequence item 1: expected str instance, int found

Traceback:
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 592, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 449, in __authn_helper
    if func(imapobj):
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 362, in __authn_gssapi
    imapobj.authenticate('GSSAPI', self.__gsshandler)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command
    literal = literator(data, rqb)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process
    ret = self.mech(self.decode(data))
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 318, in __gsshandler
    reply = ''.join(reply

The following patch has fixed the issue for the reporter of one of the Debian bug.

diff --git a/offlineimap/imapserver.py b/offlineimap/imapserver.py
index 3970541..bd377f7 100644
--- a/offlineimap/imapserver.py
+++ b/offlineimap/imapserver.py
@@ -310,12 +310,8 @@ class IMAPServer:
             # This is a behavior we got from pykerberos.  First byte is one,
             # first four bytes are preserved (pykerberos calls this a length).
             # Any additional bytes are username.
-            reply = []
-            reply[0:4] = response.message[0:4]
-            reply[0] = '\x01'
-            if self.username:
-                reply[5:] = self.username
-            reply = ''.join(reply)
+            reply = b'\x01' + response.message[1:4]
+            reply += bytes(self.username, 'utf-8')

             response = self.gss_vc.wrap(reply, response.encrypted)
             return response.message if response.message else ""

Error when creating folders with spaces on GMail's IMAP server

General informations

system/distribution (with version): Arch Linux, kernel 5.11.1-arch1-1
offlineimap version (offlineimap -V):
- from offlineimap -V: offlineimap v7.3.0, imaplib2 v3.06, Python v3.9.2, OpenSSL 1.1.1j 16 Feb 2021
- it's actually offlineimap3 commit 1e7ef9e with patches from #55 and changes according to jazzband/imaplib2#15)
Python version: 3.9
server name or domain: gmail
CLI options: -o -a test -c ${HOME}/config.test

Configuration file offlineimaprc

[general]
metadata = $HOME/offlineimap_tmp
accounts = test

[Account test]
localrepository = test-local
remoterepository = test-remote

[Repository test-local]
type = GmailMaildir
localfolders = $HOME/offlineimap_test
sep = .

[Repository test-remote]
type = Gmail
remoteuser = <account>
auth_mechanisms = XOAUTH2
oauth2_client_id = <id>
oauth2_client_secret = <secret>
oauth2_access_token = <token>
folderfilter = lambda folder: folder  == "Test ABC"

Logs, error

OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.06, Python v3.9.2, OpenSSL 1.1.1j  16 Feb 2021
Account sync test:
 *** Processing account test
 Establishing connection to imap.gmail.com:993 (test-remote)
 Creating folder Test ABC[test-remote]
 ERROR: While attempting to sync account 'test'
  CREATE command error: BAD [b'Could not parse command']. Data: b'HENK8 CREATE Test ABC\r\n'
 *** Finished account 'test' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'test'
  CREATE command error: BAD [b'Could not parse command']. Data: b'HENK8 CREATE Test ABC\r\n'

Traceback:
  File "/usr/lib/python3.9/site-packages/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/lib/python3.9/site-packages/offlineimap/accounts.py", line 377, in __sync
    remoterepos.sync_folder_structure(localrepos, statusrepos)
  File "/usr/lib/python3.9/site-packages/offlineimap/repository/Base.py", line 349, in sync_folder_structure
    remote_repo.makefolder(remote_name)
  File "/usr/lib/python3.9/site-packages/offlineimap/repository/IMAP.py", line 816, in makefolder
    self.makefolder_single(folder_path)
  File "/usr/lib/python3.9/site-packages/offlineimap/repository/IMAP.py", line 839, in makefolder_single
    result = imapobj.create(foldername)
  File "/usr/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 767, in create
    return self._simple_command('CREATE', mailbox, **kw)
  File "/usr/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 1705, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 1452, in _command_complete
    raise self.error('%s command error: %s %s. Data: %.100s' % (rqb.name, typ, dat, rqb.data))

Steps to reproduce the error

Step 1: create a test maildir

$ mkdir -p \
    $HOME/offlineimap_test/Test\ ABC/cur
    $HOME/offlineimap_test/Test\ ABC/new
    $HOME/offlineimap_test/Test\ ABC/tmp

Step 2: run offlineimap

$ offlineimap -o -a test -c ${HOME}/config.test

Description

When offlineimap tried to create a folder on GMail's IMAP server, and when that folder had spaces in its folder name, the creation failed.

Temporary solution

I have a workaround on my machine: wrapping folder names with double-quotes. However, I believe this workaround is not robust. If a folder name already has double-quotes in its name, I don't know if this workaround is going to work (though I doubt how many people have quotation marks in their folders' names). Here's my workaround:

Replace line 839 in repository/IMAP.py:

offlineimap3/offlineimap/repository/IMAP.py

Line 839 in 1e7ef9e

result = imapobj.create(foldername)

with

result = imapobj.create("\"{}\"".format(foldername))

error code handling seems to be wrong in imaplibutil.py

General informations

system/distribution (with version): Fedora 34
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.06, Python v3.9.4, OpenSSL 1.1.1k FIPS 25 Mar 2021
$ rpm -qi offlineimap
Name : offlineimap
Version : 7.3.3
Release : 6.20200310git1e7ef9e7e6952f5d29ef0f5c25fd062798de55f3.fc34
Python version: Python 3.9.4

I tried to connect with offlineimap to an onion v2 service via latest tor alpha release which was refused by tor (since v2 onion services will be dropped).

This rejected the socket, but the error message I got was rather confusing, actually it was rather a python error message than the intended one about the socket.

It looks like the error handling code in imaplibutil is not correct, but I lack enough python to fully understand.

This issue is mainly about pointing out the wrong error handling.

Account sync fooo:
*** Processing account fooo
Establishing connection to foo.onion:993 (foo_r)
1619700555 ERROR torsocks[69966]: General SOCKS server failure (in socks5_recv_connect_reply() at socks5.c:527)
ERROR: While attempting to sync account 'foo'
local variable 'msg' referenced before assignment
*** Finished account 'foo' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'foo'
local variable 'msg' referenced before assignment

Traceback:
File "/usr/lib/python3.9/site-packages/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
File "/usr/lib/python3.9/site-packages/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
File "/usr/lib/python3.9/site-packages/offlineimap/repository/IMAP.py", line 672, in getfolders
imapobj = self.imapserver.acquireconnection()
File "/usr/lib/python3.9/site-packages/offlineimap/imapserver.py", line 563, in acquireconnection
imapobj = imaplibutil.WrappedIMAP4_SSL(
File "/usr/lib/python3.9/site-packages/offlineimap/imaplibutil.py", line 192, in init
super(WrappedIMAP4_SSL, self).init(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/offlineimap/imaplib2.py", line 2124, in init
IMAP4.init(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)
File "/usr/lib/python3.9/site-packages/offlineimap/imaplib2.py", line 357, in init
self.open(host, port)
File "/usr/lib/python3.9/site-packages/offlineimap/imaplibutil.py", line 200, in open
super(WrappedIMAP4_SSL, self).open(host, port)
File "/usr/lib/python3.9/site-packages/offlineimap/imaplib2.py", line 2136, in open
self.sock = self.open_socket()
File "/usr/lib/python3.9/site-packages/offlineimap/imaplibutil.py", line 101, in open_socket
raise socket.error(msg)

As a reference simple openssl:

$ echo | torsocks openssl s_client -servername autinv5q6en4gpf4.onion -connect autinv5q6en4gpf4.onion:993
1619701549 ERROR torsocks[72533]: General SOCKS server failure (in socks5_recv_connect_reply() at socks5.c:527)
139638785414976:error:0200206F:system library:connect:Connection refused:crypto/bio/b_sock2.c:110:
139638785414976:error:2008A067:BIO routines:BIO_connect:connect error:crypto/bio/b_sock2.c:111:
connect:errno=111

For reference: https://lists.torproject.org/pipermail/tor-talk/2021-March/045716.html

'int' object is not subscriptable: connecting to outlook.office365.com:993

General informations

system/distribution (with version): Debian testing
offlineimap version (offlineimap -V): offlineimap v8.0.0, imaplib2 v3.05, Python v3.9.7, OpenSSL 1.1.1l 24 Aug 2021
Python version: above
server name or domain: outlook.office365.com as below & above
CLI options: Just -a as below

Configuration file offlineimaprc

[Repository oRemote]
type = IMAP
ssl = yes
ssl_version = tls1
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
remoteprt = 993
remotehost = outlook.office365.com

remoteuser = xxxxxxxxxx
folderfilter = lambda foldername: foldername in ['INBOX', 'Junk Email', 'Sent Items']



- offlineimap -a account
OfflineIMAP 8.0.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.9.7, OpenSSL 1.1.1l  24 Aug 2021
Account sync oucs:
 *** Processing account oucs
 Establishing connection to outlook.office365.com:993 (oRemote)
 ERROR: While attemptng to sync account 'oucs'
  'int' object is not subscriptable
 *** Finished account 'account' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'account'
  'int' object is not subscriptable

Traceback:
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 681, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 683, in acquireconnection
    e.args[0][:35] == 'IMAP4 protocol error: socket error:':
----------------------------------------------------------------------------------------

Sorry. Not looked into this any further as yet. I suppose I should build the current version and check, but a very quick skim of the git log didn't suggest anything.

Note: it seems to be nondeterministic, although very solid at times. Maybe some odd race condition?



-

syncing custom imap keywords to remote

General informations

system/distribution (with version): Fedora 5.13.12-200.fc34.x86_64
offlineimap version (offlineimap -V): v7.3.0
Python version: v3.9.6

Configuration file offlineimaprc

[general]
accounts = private

[Account private]
localrepository = plocal
remoterepository = premote

[Repository plocal]
type = Maildir
localfolders = ~/imail/private
customflag_a = my_keyword

[Repository premote]
type = IMAP
remotehost = ...
remoteuser = ...
remotepass = ...
readonly = False

Hi, I am experimenting with the customflag feature for IMAP keywords (not Gmail tags) that is
explained from line 612 on in offlineimap.conf. It has nicely worked to get keywords from my IMAP mailbox
to my local mail repository.
I was wondering if and how I can use this to pass keywords to untagged messages and then send these information back to the server. I made a naive attempt and renamed the file corresponding to the email, i.e.
16306489...66bd4c0b013af27acac3:2,S -> 16306489...66bd4c0b013af27acac3:2,Sa
This was recognized when I ran offlineimap afterwards:
Adding flag a to 1 messages on folder
But I couldn't see the added keyword in other mail programs.

I have also found this old wiki entry but am not sure how it relates to editing the flags in the configuration file.

Unable to run setup.py from a clean checkout

General informations

system/distribution (with version): macOX
offlineimap version (offlineimap -V): N/A
Python version: 3.8.3
server name or domain: N/A
CLI options: N/A

Repro steps

$ gh repo clone OfflineIMAP/offlineimap3
...
# create a virtualenv for this project here
$ pip freeze
# empty
$ pip install -r requirements.txt
...
$ pip install -r tests/requirements.txt
...
$ pip freeze
attrs==20.3.0
certifi==2020.12.5
chardet==4.0.0
codecov==2.1.11
coverage==5.3.1
decorator==4.4.2
distro==1.5.0
gssapi==1.6.12
idna==2.10
iniconfig==1.1.1
packaging==20.8
pluggy==0.13.1
portalocker==2.1.0
py==1.10.0
pyparsing==2.4.7
pytest==6.2.1
pytest-cov==2.11.1
requests==2.25.1
rfc6555==0.0.0
selectors2==2.0.2
toml==0.10.2
urllib3==1.26.2
$ python setup.py build
Traceback (most recent call last):
  File "setup.py", line 25, in <module>
    import offlineimap
  File "/Users/offby1/projects/offlineimap3/offlineimap/__init__.py", line 20, in <module>
    from offlineimap.init import OfflineImap
  File "/Users/offby1/projects/offlineimap3/offlineimap/init.py", line 29, in <module>
    import imaplib2 as imaplib
ModuleNotFoundError: No module named 'imaplib2'

So, uh, how do I get started here? I'm interested in contributing, but it seems that there's something just missing here. My first impulse was to install imaplib2, but that gives me an error that clearly indicates that imaplib2 doesn't support Python 3:

$ pip install imaplib2
...
$ python setup.py build
...
  File "/Users/offby1/projects/offlineimap3/offlineimap/imaplibutil.py", line 30, in <module>
    from imaplib2 import IMAP4, IMAP4_SSL, InternalDate
ImportError: cannot import name 'IMAP4' from 'imaplib2' (/Users/offby1/projects/offlineimap3/.direnv/python-3.8.3/lib/python3.8/site-packages/imaplib2/__init__.py)

And trying to install the version that is called out in the README is not possible:

$ pip install 'imaplib2>=3.5'
Collecting imaplib2>=3.5
  ERROR: Could not find a version that satisfies the requirement imaplib2>=3.5 (from versions: 2.28.1, 2.28.2, 2.28.3, 2.37.4, 2.38.0, 2.43.0, 2.45.0)

key: expected bytes or bytearray, but got 'str'

I'm not sure what's causing this ?

OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.8.5, OpenSSL 1.1.1f  31 Mar 2020
Account sync [email protected]:
 *** Processing account [email protected]
 Establishing connection to imap.googlemail.com:993 (remote)
 Establishing connection to 127.0.0.1:143 (local)
 ERROR: While attempting to sync account [email protected]'
  key: expected bytes or bytearray, but got 'str'
 *** Finished account '[email protected]' in 0:04
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account '[email protected]'
  key: expected bytes or bytearray, but got 'str'

Traceback:
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/accounts.py", line 375, in __sync
    localrepos.getfolders()
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/repository/IMAP.py", line 451, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/imapserver.py", line 579, in acquireconnection
    self.__authn_helper(imapobj)
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/imapserver.py", line 443, in __authn_helper
    if func(imapobj):
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/imapserver.py", line 365, in __authn_cram_md5
    imapobj.authenticate('CRAM-MD5', self.__md5handler)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command
    literal = literator(data, rqb)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process
    ret = self.mech(self.decode(data))
  File "/usr/store1/usr/local/src/offlineimap/offlineimap3-master/offlineimap/imapserver.py", line 187, in __md5handler
    retval = self.username + ' ' + hmac.new(passwd, challenge).hexdigest()
  File "/usr/lib/python3.8/hmac.py", line 153, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib/python3.8/hmac.py", line 48, in __init__
    raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__)

Issues from offlineimap (py2) to solve in offlineimap (py3)

Bugs opened in offlineimap (Python2) and their status in offlineipam (Python 3)

NOTES:

Do not post comments about these bugs here. Open a new bug we will link it here.
Please, update this table and try to solve these bugs.

kix

Solved	Level	Title	Assigned	Notes	Links
		Store files with an .eml extension
		CRAM-MD5 authentication failed type 'exceptions.IOError'
		How to use the collected budget
		ERROR: source object is already set
		Gmail and Labels: label correspondig to folder not added to header?		Gmail
		XOAUTH2: xoauth2handler got: {u'error_description': u'Bad Request', u'error': u'invalid_grant'}
Yes		Support Python 3	need contributor	Py3, This version is written in Python3
	feature	No enforcement of STARTTLS.	need contributor
	feature	Replicate delete folders from remote to local copy
		OAuth2 connection fails only for refresh token
	feature	Handle server side throttling	need contributor	good first issue
		Synchronize Keywords
	bug	don't assume the sep character is always required	need contributor
		When pushing old mails to Gmail, Gmail shows sync date rather than original date	need contributor	documentation, Gmail
	bug	--delete-folder with the blinkenlight UI results in a borked terminal.	need contributor
		Failed to do SSL handshake when using proxy
	bug	seems to stall when syncing with gmail	need contributor	Gmail
	feature	passwords should not be expanded in configuration	need contributor	good first issue
		Login strange log failure
	question	Login with IMAP-Password with special chars impossible
	question	How to specify log file in .offlineimaprc ? It's undocumented, but obvious that is needed
		Regression in v7.2.4: creating new (sub)folders fails
		Issue with localized names
	bug	How to deal with ERROR messages	need contributor
	bug	Big email is downloaded despite exceeding size specified in maxsize option	need contributor	good first issue
	feature	idlefolders as a filter	need contributor
	bug	GMail folders starting with '.' (dot) do not get synced		Gmail
	bug	SSL: CERTIFICATE_VERIFY_FAILED to Exchange Server
	feature	RC service	need contributor
	bug	Renamed gmail folders are skipped	need contributor	Gmail
	bug	Crash on remote folder cur, new and tmp	need contributor
	bug	Crash on subfolder	need contributor
	bug	KeyError in `folder/Gmail.py`		Gmail
		Unable to sync INBOX folder
		IMAP4 protocol error: socket error: <type 'exceptions.IOError'>	need contributor
		Maildir storage of folders violates the standard		wontfix
	bug	Zero byte message synced	need contributor
	bug	CERTIFICATE_VERIFY_FAILED but openssl s_client works with same ca certs file	need contributor	Gmail
	feature	TLS 1.3	need contributor	good first issue
	bug	Traceback during docu generation		documentation
	enhancement	indented comments confuse config parser	need contributor	documentation
	bug	-u syslog doesn't work on macOS	need contributor	MacOS
	CRITICAL	Does not check SSL Certificate Revocation Status	need contributor
	enhancement	XOAUTH2 authentication failed	need contributor
	feature	introduce a configuration option to define the internaldate used for APPEND	need contributor	good first issue
	bug	netrc cannot contain spaces in password	need contribturo	upstream
	enhancement	Include exit code of pre/post sync hooks when exiting?	need contributor	good first issue
	enhancement	offlineimap cannot acquire lock after system suspend	need contributor	good first issue
	feature	MOVE command	need contributor
	feature	Add Autoconfig, Autodiscover. Import exchangelib for Microsoft compatibility.		Microsoft
	bug	Email from office365.com with event breaks offlineimap	need contributor
	feature	handle enabling/disabling utf8foldernames nicely
	feature	Script in cert_fingerprint	need contributor
	feature	drying config file	need contributor
	enhancement	Password printed when "All authentication types failed"	need contributor
	bug	SSL: no shared cipher	need contributor
	feature	Office 365: auto-configure folderfilter by default	need contributor	Microsoft
	bug	"ValueError: too many values to unpack" when syncing from Exchange 2007	need contributor	Microsoft	x
	enhancement	Automatic filtering of special folders (Calendar, etc)	need contributor	Microsoft
	bug	subscribedonly tries to recreate existing folder when folder is unsubscribed	need contributor
	CRITICAL	(Data loss) Issue with symlink causes entire remote IMAP store to be purged/deleted.
	feature	Keywords: allow users to configure the separator	need contributor
	enhancement	Explore possibility of using IMAPClient	need contributor
	bug	Big messages cause an APPEND timeout	need contributor
	enhancement	changing nametrans on a local folder invalidates all uids.	need contributor	good first issue
	feature	support checking the database files	need contributor	good first issue
	bug	OfflineImap leaks (or retains) a lot of memory autoreshresh	need contributor
	bug	Primary Key must be unique error IDLE	need contributor
Yes	bug	Cannot use remotepasseval with Python3	need contributor	Py3, this should not happen in Python 3	original
	feature	introduce hook to track down any change to local Maildirs	need contributor	good first issue
Yes	bug	Too many read 0, while fetching msg '214' Py3	need contributor	Py3, this bug should not happen in Python 3	original
	feature	Support user flags	need contributor
	bug	MemoryError on RaspberryPi	need contributor
	enhancement	optimize re-download on UIDVALIDITY changes	need contributor
	feature	open_socket() leaks DNS requests when proxying	need contributor
	feature	Add IMAP NOTIFY support.	need contributor
	bug	Unexpected hangs on timeouts (system resume)	need contributor
	enhancement	Failure when files changed between invokation and actual sync.	need contributor
	feature	Support the QRESYNC extension	need contributor
Yes	feature	Automagically fetch Gmail certificates	need contributor	Gmail	original Patch
	enhancement	Attempting NOOP on dropped connection	need contributor
	bug	offlineimap confused after (suspend and) resume	need contributor
	enhancement	--dry-run can drive you nuts	need contributor

exception on mailboxes with spaces in name

General informations

system/distribution (with version): Gentoo Linux -- latest 2020-09-26
offlineimap version (offlineimap -V): git commit 8dca916 (output printed: offlineimap v7.3.0, imaplib2 v3.05, Python v3.7.8, OpenSSL 1.1.1g 21 Apr 2020)
Python version: 3.7
server name or domain: /
CLI options: none

Logs, error

Folder foo bar [acc: my-account]:
 ERROR: ERROR in syncfolder for my-account folder foo bar: Traceback (most recent call last):
  File "/home/user/repo/offlineimap3/offlineimap/accounts.py", line 651, in syncfolder
    check_uid_validity()
  File "/home/user/repo/offlineimap3/offlineimap/accounts.py", line 512, in check_uid_validity
    if not remotefolder.check_uidvalidity():
  File "/home/user/repo/offlineimap3/offlineimap/folder/Base.py", line 227, in check_uidvalidity
    return self.get_saveduidvalidity() == self.get_uidvalidity()
  File "/home/user/repo/offlineimap3/offlineimap/folder/IMAP.py", line 138, in get_uidvalidity
    self.__selectro(imapobj)
  File "/home/user/repo/offlineimap3/offlineimap/folder/IMAP.py", line 86, in __selectro
    imapobj.select(self.getfullIMAPname(), force=force)
  File "/home/user/repo/offlineimap3/offlineimap/imaplibutil.py", line 51, in select
    result = super(UsefulIMAPMixIn, self).select(mailbox, readonly)
  File "/home/user/repo/offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1053, in select
    self._deliver_exc(self.error, '%s command error: %s %s. Data: %.100s' % (name, typ, dat, mailbox), kw)
  File "/home/user/repo/offlineimap3/venv/lib/python3.7/site-packages/imaplib2.py", line 1467, in _deliver_exc
    raise exc(dat)
imaplib2.IMAP4.error: SELECT command error: BAD [b'Command Argument Error. 12']. Data: foo bar

  SELECT command error: BAD [b'Command Argument Error. 12']. Data: foo bar

Steps to reproduce the error

This error occurs always, on mailboxes that have a space in the name, like 'foo bar'.

I can fix/workaround the problem by changing imaplib2.py as follows:

--- venv/lib/python3.7/site-packages/imaplib2.py.orig	2020-09-28 11:05:07.235030053 +0200
+++ venv/lib/python3.7/site-packages/imaplib2.py	2020-09-28 11:05:13.925078829 +0200
@@ -1043,7 +1043,7 @@
         else:
             name = 'SELECT'
         try:
-            rqb = self._command(name, mailbox)
+            rqb = self._command(name, self._quote(mailbox))
             typ, dat = rqb.get_response('command: %s => %%s' % rqb.name)
             if typ != 'OK':
                 if self.state == SELECTED:

i.e. quoting the mailbox name, as specified by the IMAP4 RFC.

However, the above fix does not seem complete (there may be other places where the same quoting should happen).
Moreover, when using offlineimap for Python 2 (version: "offlineimap v7.3.3, imaplib2 v2.101 (bundled), Python v2.7.18, OpenSSL 1.1.1g 21 Apr 2020" ), this problem does not occur.

offlineimap3: fails with "'str' object has no attribute 'decode'"

Using nametrans leads to a failure with offlineimap3 but works with offlineimap.

   ERROR: While attempting to sync account 'testmail'
      'str' object has no attribute 'decode'
    
    Traceback:
      File "/usr/lib/python3/dist-packages/offlineimap/accounts.py", line 298, in syncrunner
        self.__sync()
      File "/usr/lib/python3/dist-packages/offlineimap/accounts.py", line 374, in __sync
        remoterepos.getfolders()
      File "/usr/lib/python3/dist-packages/offlineimap/repository/IMAP.py", line 679, in getfolders
        retval.append(self.getfoldertype()(self.imapserver, name,
      File "/usr/lib/python3/dist-packages/offlineimap/folder/IMAP.py", line 53, in __init__
        super(IMAPFolder, self).__init__(name, repository)
      File "/usr/lib/python3/dist-packages/offlineimap/folder/Base.py", line 57, in __init__
        self.visiblename = repository.nametrans(name)
      File "<string>", line 1, in <lambda>

Steps to reporduce:

extract the attached file and use it as pythonfile in [general] of .offlineimaprc.
Add nametrans to your .offlineimaprc.
nametrans = lambda foldername: foldername.decode('imap4-utf-7').encode('utf-8')

offlineimap.zip

offlineimap error upon process start - new installation of offlineimap(3) on debian 11

General informations

system/distribution (with version): Debian GNU/Linux 11 (bullseye)
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.2, OpenSSL 1.1.1k 25 Mar 2021
offlineimap pkgs (thru dpkg-query -l):

ii  offlineimap                           7.3.3+dfsg1-1+0.0\~git20210225.1e7ef9e+dfsg-4 all          transitional package
ii  offlineimap3                          0.0~git20210225.1e7ef9e+dfsg-4               all          IMAP/Maildir synchronization and reader support
ii  python3-imaplib2                      2.57-5.2                                     all          Threaded Python IMAP4 client (Python 3)

Python version: 3.9.2
server name or domain: https://accounts.google.com/o/oauth2/token
CLI options: none

Configuration file offlineimaprc

[general]
metadata = ~/.offlineimap
accounts = gmail
maxsyncaccounts = 1
pythonfile = $XDG_CONFIG_HOME/offlineimap/.offlineimap.py
socktimeout = 60

[Account gmail]
localrepository = LocalGmail
remoterepository = RemoteGmail
quick = 10
maxsize = 2000000
maxage = 2015-04-01
#maxage = 10
postsynchook = mu index --maildir ~/mails

[Repository LocalGmail]
type = Maildir
localfolders = ~/mails/gmail

[Repository RemoteGmail]
type = Gmail
maxconnections = 2
remoteuser = [email protected]

oauth2_request_url = https://accounts.google.com/o/oauth2/token
oauth2_client_id_eval = mailpasswd("gmail/oauth2_client_id")
oauth2_client_secret_eval = mailpasswd("gmail/oauth2_client_secret")
oauth2_refresh_token_eval = mailpasswd("gmail/oauth2_refresh_token")

folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail', '[Gmail]/Important']
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

pythonfile (if any)

import os
import subprocess

def mailpasswd(acct):
  # acct = os.path.basename(acct)
  path = "/home/xxx/.password-store/%s.gpg" % acct
  args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
  try:
    return subprocess.check_output(args).strip()
  except subprocess.CalledProcessError:
    return ""

def prime_gpg_agent():
  ret = False
  i = 1
  while not ret:
    ret = (mailpasswd("generic/prime") == "prime")
    if i > 2:
      from offlineimap.ui import getglobalui
      sys.stderr.write("Error reading in passwords. Terminating.\n")
      getglobalui().terminate()
    i += 1
  return ret

prime_gpg_agent()

Logs, error

Traceback (most recent call last):
  File "/usr/bin/offlineimap", line 22, in <module>
    oi.run()
  File "/usr/share/offlineimap3/offlineimap/init.py", line 86, in run
    options, args = self.__parse_cmd_options()
  File "/usr/share/offlineimap3/offlineimap/init.py", line 262, in __parse_cmd_options
    config.set_if_not_exists('general', 'dry-run', 'False')
  File "/usr/share/offlineimap3/offlineimap/CustomConfig.py", line 127, in set_if_not_exists
    self.set(section, option, value)
  File "/usr/lib/python3.9/configparser.py", line 1201, in set
    super().set(section, option, value)
  File "/usr/lib/python3.9/configparser.py", line 902, in set
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'general'

Steps to reproduce the error

offlineimap on command line, w/o any options.

Other comments

The error happens very early on after launching offlineimap process; from what I can tell the parser is not able unable to find the [general] section in offlineimap config file. My setup appears to be correct, but happy to get feedback.

Also, would be very helpful to get confirmation if the pythonfile script is Python 3 compatible.

Thank You in advance!

CRAM-MD5 authentication broken - encoding without a string argument

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): 0.0~git20210105.00d395b+dfsg-2
Debian Bug: https://bugs.debian.org/981385

Logs, error

2021-01-30 14:02:59 ERROR: ERROR: While attempting to sync account 'accountname'
  encoding without a string argument
2021-01-30 14:02:59 ERROR: ['  
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner\n    self.__sync()\n', '  
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync\n    remoterepos.getfolders()\n', '  
File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders\n    imapobj = self.imapserver.acquireconnection()\n', ' 
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 592, in acquireconnection\n    self.__authn_helper(imapobj)\n', '  
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 449, in __authn_helper\n    if func(imapobj):\n', '  
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 371, in __authn_cram_md5\n    imapobj.authenticate(\'CRAM-MD5\', self.__md5handler)\n', '  
File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate\n    typ, dat = self._simple_command(\'AUTHENTICATE\', mechanism.upper())\n', '  
File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command\n    return self._command_complete(self._command(name, *args), kw)\n', '  
File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command\n    literal = literator(data, rqb)\n', '  
File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process\n    ret = self.mech(self.decode(data))\n', '  
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 190, in __md5handler\n    hmac.new(bytes(passwd, encoding=\'utf-8\'), challenge,\n']

Steps to reproduce the error

Non-UTF-8 attachments are messed up after being downloaded by Offlineimap3

I am not entirely sure this is indeed an Offlineimap3 bug. But I upgraded to offlineimap3 yesterday evening, and after starting it for the first time this morning, I had to correct my python script that decodes with GPG the files where I store my passwords, and now this script is working but I still have some weird behavior with my emails that did not exist yesterday. Since the python script/GPG thing was clearly related to Offlineimap3, I am guessing that this is not a coincidence and Offlineimap3 is also responsible of the other bug. Moreover, the emails that are messed up on my Debian using offlineimap are not messed up on my phone using K9-Mail, so I’d say it is not the emails themselves that have a problem, but indeed my Debian setup (so, mutt+offlineimap).

The problem is that emails with non-utf-8 charset, for instance iso-8859-15, that have been downloaded by Offlineimap3 (so, downloaded today) are not displayed correctly. This is the case both for text/html attachments (displayed in mutt with w3m) or for text/plain attachments. I have strings like ï¿œ everywhere I should have é, for instance.

Similar emails that were downloaded before the Offlineimap3 upgrade are displayed correctly, so the problem really seems to be with Offlineimap3.

If I run file on one of these text attachments, I find that the charset is UTF-8, even if the HTML header says otherwise. For emails downloaded before the Offlineimap3 upgrade, the charset indicated by file and that indicated in the HTML header are consistent.

If I run iconv -f UTF-8 -t UTF-8 on one of these text attachments, the ï¿œ is changed into �.

So my feeling is that somehow Offlineimap3 writes all emails it is downloading in UTF-8 but does not convert the characters from the old charset to the UTF-8 charset properly. By doing so, characters like é are changed into the special UTF-8 character �, but then mutt and w3m still think that the file is in ISO-8859 or whatever because this is what is written in the HTML header, and are unable to display correctly the text.

I am not a charset expert and I am probably not making any sense, sorry. I will try to answer as precisely as I can any question you might have.

General informations

system/distribution (with version): Debian testing
offlineimap version (offlineimap -V): 7.3.0
Python version: 3.9.1
server name or domain:
CLI options:

Configuration file offlineimaprc

[general]
# General information.  See the fully annotated example more information
# https://github.com/jgoerzen/offlineimap/blob/master/offlineimap.conf
pythonfile = ~/Programmation/Scripts/offlineimap_gpg.py

metadata = ~/.offlineimap
accounts = account1,account2,account3,account4
maxsyncaccounts = 6
socktimeout = 60
ui = basic

[mbnames]
# This populates a mailbox file for mutt to use.
# We disabled this once it was made !

enabled = no
filename = ~/.mail_configs/mutt/mailboxes
header = "mailboxes "

peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

[Account account1]
localrepository = local-account1
remoterepository = remote-account1
autorefresh = 2
quick = 2
postsynchook = ~/Programmation/Scripts/spamscanner.sh

[Account account2]
localrepository = local-account2
remoterepository = remote-account2
autorefresh = 2
quick = 2

[Account account3]
localrepository = local-account3
remoterepository = remote-account3
autorefresh = 2
quick = 2

[Account account4]
localrepository = local-account4
remoterepository = remote-account4
autorefresh = 2
quick = 2

[Repository local-account1]
type = Maildir
localfolders = /home/elvith/.mail/account1
nametrans = lambda foldername: re.sub ('INBOX.INBOX', 'INBOX', re.sub (r'^', r'INBOX.', foldername))

[Repository remote-account1] 
type = IMAP 
remotehost = REDACTED
remoteuser = REDACTED
remotepasseval = get_pass_account1()
ssl = yes
sslcacertfile = OS-DEFAULT
realdelete = yes
maxconnections = 3
#holdconnectionopen = true
#keepalive = 60
folderfilter = lambda f: f not in ['Templates']
nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)

[Repository local-account2]
type = Maildir
localfolders = /home/elvith/.mail/account2
nametrans = lambda foldername: re.sub ('inbox', 'INBOX', foldername) 

[Repository remote-account2] 
type = IMAP 
remotehost = REDACTED
remoteuser = REDACTED
remotepasseval = get_pass_account2()
ssl = yes
sslcacertfile = OS-DEFAULT
realdelete = yes
maxconnections = 3
#holdconnectionopen = true
#keepalive = 60
nametrans = lambda foldername: re.sub ('INBOX', 'inbox', foldername)

[Repository local-account3]
type = Maildir
localfolders = /home/elvith/.mail/account3 
nametrans = lambda foldername: re.sub ('inbox', 'INBOX', foldername) 

[Repository remote-account3]
type = IMAP
remotehost = REDACTED
remoteuser = REDACTED
remotepasseval = get_pass_account3()
ssl = yes
sslcacertfile = OS-DEFAULT
ssl_version = tls1_2
cert_fingerprint = 4169f501ed5ddfcc6c6705bc9114bae5c89b8ca7
realdelete = yes
maxconnections = 3
#holdconnectionopen = true
#keepalive = 60
folderfilter = lambda f: f in ['INBOX','sent','trash']
nametrans = lambda foldername: re.sub ('INBOX', 'inbox', foldername)

[Repository local-account4]
type = Maildir
localfolders = /home/elvith/.mail/account4
nametrans = lambda foldername: re.sub ('INBOX.INBOX', 'INBOX', re.sub ('inbox', 'INBOX', re.sub (r'^', r'INBOX.', foldername)))

[Repository remote-account4]
type = IMAP
remotehost = REDACTED
remoteuser = REDACTED
remotepasseval = get_pass_account4()
ssl = yes
sslcacertfile = OS-DEFAULT
realdelete = yes
maxconnections = 3
#holdconnectionopen = true
#keepalive = 60
folderfilter = lambda foldername: not re.search('INBOX.INBOX.', foldername)

# Change INBOX.*, and lowercase
nametrans = lambda foldername: re.sub('^inbox\.', '', foldername.lower())

pythonfile (if any)

This is the file offlineimap_gpg.py that is referred to in the file .offlineimaprc above

#! /usr/bin/env python
from subprocess import check_output

def get_pass_account1():
        return check_output("gpg -dq ~/Documents/Informatique/password_account1.gpg", shell=True).decode().strip("\n")

def get_pass_account2():
        return check_output("gpg -dq ~/Documents/Informatique/password_account2.gpg", shell=True).decode().strip("\n")

def get_pass_account3():
        return check_output("gpg -dq ~/Documents/Informatique/password_account3.gpg", shell=True).decode().strip("\n")

def get_pass_account4():
        return check_output("gpg -dq ~/Documents/Informatique/password_account4.gpg", shell=True).decode().strip("\n")

Authentication fails while using Curses

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): 0.0~git20210105.00d395b+dfsg-2
Debian bugs: https://bugs.debian.org/981949 and https://bugs.debian.org/981960

Configuration file offlineimaprc

I have asked for the offlineimaprc from the reporter.

Logs, error

2021-02-05 11:14:15 INFO: OfflineImap 7.3.0 starting...
  Python: 3.9.1 Platform: linux
  Args: /usr/bin/offlineimap -a Example -d ALL -l /tmp/offlineimap.log
2021-02-05 11:14:15 INFO: OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
2021-02-05 11:14:15 INFO: imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i  8 Dec 2020
2021-02-05 11:14:15 DEBUG: Now debugging for imap: IMAP protocol debugging
2021-02-05 11:14:15 DEBUG: Now debugging for maildir: Maildir repository debugging
2021-02-05 11:14:15 DEBUG: Now debugging for thread: Threading debugging
2021-02-05 11:14:15 DEBUG: Now debugging for : Other offlineimap related sync messages
2021-02-05 11:14:15 DEBUG: [thread]: Register new thread 'Account sync Example' (account 'Example')
2021-02-05 11:14:15 DEBUG: [imap]: Using authentication mechanisms ['GSSAPI', 'XOAUTH2', 'CRAM-MD5', 'PLAIN', 'LOGIN']
2021-02-05 11:14:15 DEBUG: [maildir]: MaildirRepository initialized, sep is '.'
2021-02-05 11:14:15 INFO: *** Processing account Example
2021-02-05 11:14:15 INFO: Establishing connection to imap.example.net:993 (LPB-Remote)
2021-02-05 11:14:15 DEBUG: [imap]: LPB-Remote: level 'tls_compat', version 'None'
2021-02-05 11:14:15 DEBUG: [imap]:   14:15.82 Account sync Example imaplib2 version 3.05
2021-02-05 11:14:15 DEBUG: [imap]:   14:15.82 Account sync Example imaplib2 debug level 5, buffer level 3
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.05 Account sync Example connected to imap.example.net on port 993
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.05 imap.example.net writer starting
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.05 imap.example.net reader starting using poll
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.06 Account sync Example _request_push(continuation, welcome, {}) = b'CJOD0'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.06 Account sync Example welcome:b'CJOD0'.ready.wait
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.09 imap.example.net reader poll => [(5, 1)]
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.10 imap.example.net reader rcvd 18
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.10 imap.example.net reader < b'* OK IMAP4 ready\r\n'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.15 imap.example.net handler starting
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 imap.example.net handler _put_response(b'* OK IMAP4 ready')
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 imap.example.net handler untagged_responses[OK] 0 += ["b'IMAP4 ready'"]
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 imap.example.net handler _request_pop(continuation, (False, b'* OK IMAP4 ready')) [0] = b'CJOD0'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 imap.example.net handler welcome:b'CJOD0'.ready.set
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example _get_untagged_response(OK) => [b'IMAP4 ready']
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example state => NONAUTH
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example [async] CAPABILITY ()
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example state_change_pending.acquire
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example state_change_pending.release
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 imap.example.net handler state_change_free.set
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example _request_push(b'CJOD1', CAPABILITY, {}) = b'CJOD1'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example data=b'CJOD1 CAPABILITY'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 Account sync Example CAPABILITY:b'CJOD1'.ready.wait
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.16 imap.example.net writer > b'CJOD1 CAPABILITY\r\n'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.21 imap.example.net reader poll => [(5, 1)]
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.21 imap.example.net reader rcvd 63
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.21 imap.example.net reader < b'* CAPABILITY IMAP4rev1 UIDPLUS AUTH=PLAIN\r\n'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.21 imap.example.net reader < b'CJOD1 OK completed\r\n'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net handler _put_response(b'* CAPABILITY IMAP4rev1 UIDPLUS AUTH=PLAIN')
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net handler untagged_responses[CAPABILITY] 0 += ["b'IMAP4rev1 UIDPLUS AUTH=PLAIN'"]
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net handler _put_response(b'CJOD1 OK completed')
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net handler _request_pop(b'CJOD1', ('OK', [b'completed'])) [0] = b'CJOD1'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net handler CAPABILITY:b'CJOD1'.ready.set
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example _get_untagged_response(CAPABILITY) => [b'IMAP4rev1 UIDPLUS AUTH=PLAIN']
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net handler state_change_free.set
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example _untagged_response(OK, ?, CAPABILITY) => [b'IMAP4rev1 UIDPLUS AUTH=PLAIN']
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example CAPABILITY: ('IMAP4REV1', 'UIDPLUS', 'AUTH=PLAIN')
2021-02-05 11:14:16 DEBUG: [imap]: Attempting PLAIN authentication
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example [sync] AUTHENTICATE ('PLAIN',)
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example state_change_pending.acquire
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example _request_push(b'CJOD2', AUTHENTICATE, {}) = b'CJOD2'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example data=b'CJOD2 AUTHENTICATE PLAIN'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example _request_push(continuation, b'AUTHENTICATE', {}) = b'CJOD3'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 Account sync Example b'AUTHENTICATE':b'CJOD3'.ready.wait
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.22 imap.example.net writer > b'CJOD2 AUTHENTICATE PLAIN\r\n'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.27 imap.example.net reader poll => [(5, 1)]
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.27 imap.example.net reader rcvd 4
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.28 imap.example.net reader < b'+ \r\n'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.28 imap.example.net handler _put_response(b'+ ')
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.28 imap.example.net handler _request_pop(continuation, (True, b'')) [1] = b'CJOD3'
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.28 imap.example.net handler b'AUTHENTICATE':b'CJOD3'.ready.set
2021-02-05 11:14:16 DEBUG: [imap]:   14:16.28 Account sync Example continuation => True, b''
2021-02-05 11:14:16 WARNING:  *** Input Required
2021-02-05 11:14:16 WARNING:  *** Please enter password for user 'exampleuser': 
2021-02-05 11:14:20 DEBUG: [imap]:   14:20.76 Account sync Example state_change_pending.release
2021-02-05 11:14:20 ERROR: ERROR: While attempting to sync account 'Example'
  sequence item 2: expected str instance, bytes found
2021-02-05 11:14:20 ERROR: ['  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner\n    self.__sync()\n', '  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync\n    remoterepos.getfolders()\n', '  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders\n    imapobj = self.imapserver.acquireconnection()\n', '  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 592, in acquireconnection\n    self.__authn_helper(imapobj)\n', '  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 449, in __authn_helper\n    if func(imapobj):\n', '  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 375, in __authn_plain\n    imapobj.authenticate(\'PLAIN\', self.__plainhandler)\n', '  File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate\n    typ, dat = self._simple_command(\'AUTHENTICATE\', mechanism.upper())\n', '  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command\n    return self._command_complete(self._command(name, *args), kw)\n', '  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command\n    literal = literator(data, rqb)\n', '  File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process\n    ret = self.mech(self.decode(data))\n', '  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 217, in __plainhandler\n    retval = NULL.join((authz, authc, passwd))\n']
2021-02-05 11:14:20 INFO: *** Finished account 'Example' in 0:04

Steps to reproduce the error

Use a password which has a . in it
Add ui = blinkenlights in [general] section of .offlineimaprc
Adding remotepass or remotepassfile or asking password from ui, all will give different types of errors.

Unsafe login with 163 emails

General informations

system/distribution (with version): Void
offlineimap version (offlineimap -V): offlineimap v7.3.3, imaplib2 v2.101 (bundled), Python v2.7.18, OpenSSL 1.1.1k 25 Mar 2021
Python version: 2.7
server name or domain: imap.163.com
CLI options: none

Configuration file offlineimaprc

[Account [email protected]]
remoterepository = [email protected]_remote
localrepository = [email protected]_local

[Repository [email protected]_local]
type = Maildir
localfolders = ~/foo/bar

[Repository [email protected]_remote]
type = IMAP
remotehost = imap.163.com
remoteuser = foo
remotepass = bar
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
maxconnections = 1
readonly = True
sync_deletes = no

Logs, error

Account sync [email protected]:
 *** Processing account [email protected]
 Establishing connection to imap.163.com:993 ([email protected]_remote)
Folder &dcVr0mWHTvZZOQ- [acc: [email protected]]:
 Syncing &dcVr0mWHTvZZOQ-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&dcVr0mWHTvZZOQ-' [acc: '[email protected]']
  Error SELECTing mailbox '&dcVr0mWHTvZZOQ-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &g0l6P3ux- [acc: [email protected]]:
 Syncing &g0l6P3ux-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&g0l6P3ux-' [acc: '[email protected]']
  Error SELECTing mailbox '&g0l6P3ux-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &i6KWBZCuTvY- [acc: [email protected]]:
 Syncing &i6KWBZCuTvY-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&i6KWBZCuTvY-' [acc: '[email protected]']
  Error SELECTing mailbox '&i6KWBZCuTvY-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &V4NXPpCuTvY- [acc: [email protected]]:
 Syncing &V4NXPpCuTvY-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&V4NXPpCuTvY-' [acc: '[email protected]']
  Error SELECTing mailbox '&V4NXPpCuTvY-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &XfJSIJZk- [acc: [email protected]]:
 Syncing &XfJSIJZk-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&XfJSIJZk-' [acc: '[email protected]']
  Error SELECTing mailbox '&XfJSIJZk-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &XfJT0ZAB- [acc: [email protected]]:
 Syncing &XfJT0ZAB-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&XfJT0ZAB-' [acc: '[email protected]']
  Error SELECTing mailbox '&XfJT0ZAB-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &Xn9USpCuTvY- [acc: [email protected]]:
 Syncing &Xn9USpCuTvY-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&Xn9USpCuTvY-' [acc: '[email protected]']
  Error SELECTing mailbox '&Xn9USpCuTvY-', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder Archives [acc: [email protected]]:
 Syncing Archives: IMAP -> Maildir
 ERROR: Aborting sync, folder 'Archives' [acc: '[email protected]']
  Error SELECTing mailbox 'Archives', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder INBOX [acc: [email protected]]:
 Syncing INBOX: IMAP -> Maildir
 ERROR: Aborting sync, folder 'INBOX' [acc: '[email protected]']
  Error SELECTing mailbox 'INBOX', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])
Folder Notes [acc: [email protected]]:
 Syncing Notes: IMAP -> Maildir
 ERROR: Aborting sync, folder 'Notes' [acc: '[email protected]']
  Error SELECTing mailbox 'Notes', server reply:
('NO', ['SELECT Unsafe Login. Please contact [email protected] for help'])

Steps to reproduce the error

Just run offlineimap

Solution to the error

The IMAP server at imap.163.com implemented the IMAP ID extension. To successfully authenticate, the IMAP client must send client ID.

Here is a code snippet that I obtained from https://blog.yrpang.com/posts/45207/:

from imapclient import IMAPClient

server = IMAPClient("imap.163.com", ssl=True, port=993)
server.login("<user>", "<passwd>")

server.id_({"name": "IMAPClient", "version": "2.1.0"})

messages = server.select_folder('INBOX')

I guess this may mean that we need a configuration option that is a switch. When the switch is on, offlineimap sends IMAP client ID.

I'm posting it here because I wonder if the issue is applicable to offlineimap3 as well. If it does, it will be great if we could fix it.

Unsafe login error with imap.163.com

General informations

system/distribution (with version): Void
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.06, Python v3.9.4, OpenSSL 1.1.1k 25 Mar 2021 (Actually, it's the git master version)
Python version: 3.9
server name or domain: imap.163.com
CLI options: none

Configuration file offlineimaprc

[Account [email protected]]
remoterepository = [email protected]_remote
localrepository = [email protected]_local

[Repository [email protected]_local]
type = Maildir
localfolders = ~/foo/bar

[Repository [email protected]_remote]
type = IMAP
remotehost = imap.163.com
remoteuser = foo
remotepass = bar
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
maxconnections = 1
readonly = True
sync_deletes = no

Logs, error

OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.06, Python v3.9.4, OpenSSL 1.1.1k  25 Mar 2021
Account sync [email protected]:
 *** Processing account [email protected]
 Establishing connection to imap.163.com:993 ([email protected]_remote)
Folder &dcVr0mWHTvZZOQ- [acc: [email protected]]:
 Syncing &dcVr0mWHTvZZOQ-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&dcVr0mWHTvZZOQ-' [acc: '[email protected]']
  Error SELECTing mailbox '&dcVr0mWHTvZZOQ-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &g0l6P3ux- [acc: [email protected]]:
 Syncing &g0l6P3ux-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&g0l6P3ux-' [acc: '[email protected]']
  Error SELECTing mailbox '&g0l6P3ux-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &i6KWBZCuTvY- [acc: [email protected]]:
 Syncing &i6KWBZCuTvY-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&i6KWBZCuTvY-' [acc: '[email protected]']
  Error SELECTing mailbox '&i6KWBZCuTvY-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &V4NXPpCuTvY- [acc: [email protected]]:
 Syncing &V4NXPpCuTvY-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&V4NXPpCuTvY-' [acc: '[email protected]']
  Error SELECTing mailbox '&V4NXPpCuTvY-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &XfJSIJZk- [acc: [email protected]]:
 Syncing &XfJSIJZk-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&XfJSIJZk-' [acc: '[email protected]']
  Error SELECTing mailbox '&XfJSIJZk-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &XfJT0ZAB- [acc: [email protected]]:
 Syncing &XfJT0ZAB-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&XfJT0ZAB-' [acc: '[email protected]']
  Error SELECTing mailbox '&XfJT0ZAB-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder &Xn9USpCuTvY- [acc: [email protected]]:
 Syncing &Xn9USpCuTvY-: IMAP -> Maildir
 ERROR: Aborting sync, folder '&Xn9USpCuTvY-' [acc: '[email protected]']
  Error SELECTing mailbox '&Xn9USpCuTvY-', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder Archives [acc: [email protected]]:
 Syncing Archives: IMAP -> Maildir
 ERROR: Aborting sync, folder 'Archives' [acc: '[email protected]']
  Error SELECTing mailbox 'Archives', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder INBOX [acc: [email protected]]:
 Syncing INBOX: IMAP -> Maildir
 ERROR: Aborting sync, folder 'INBOX' [acc: '[email protected]']
  Error SELECTing mailbox 'INBOX', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Folder Notes [acc: [email protected]]:
 Syncing Notes: IMAP -> Maildir
 ERROR: Aborting sync, folder 'Notes' [acc: '[email protected]']
  Error SELECTing mailbox 'Notes', server reply:
('NO', [b'SELECT Unsafe Login. Please contact [email protected] for help'])
Account sync [email protected]:
 *** Finished account '[email protected]' in 0:05

Steps to reproduce the error

Just run offlineimap

Solution to the error

The IMAP server at imap.163.com implemented the IMAP ID extension. To successfully authenticate, the IMAP client must send client ID.

Here is a code snippet that I obtained from https://blog.yrpang.com/posts/45207/:

from imapclient import IMAPClient

server = IMAPClient("imap.163.com", ssl=True, port=993)
server.login("<user>", "<passwd>")

server.id_({"name": "IMAPClient", "version": "2.1.0"})

messages = server.select_folder('INBOX')

I guess this may mean that we need a configuration option that is a switch. When the switch is on, offlineimap sends IMAP client ID.

This error was previously reported as #68. I have tested using the latest git version of offlineimap3 and reproduced the error.

HTTP Error 400 when connecting to Gmail using XOAUTH2

General informations

system/distribution (with version): Linux Debian 11
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.2, OpenSSL 1.1.1k 25 Mar 2021
Python version: 3.9.2
server name or domain: gmail using XOAUTH2
CLI options: n/a

Configuration file offlineimaprc

see #74

pythonfile (if any)

see #74

Logs, error

ERROR: HTTP Error 400: Bad Request

Steps to reproduce the error

offlineimap on command line - no options.

Setup offlineimap last week and gmail sync was working initially, but started seeing this error since late last week. Connection to outlook email working fine.

Is this error known previously and something that may get addressed in the future as noted here:
#18

Thank You

Handle messages as bytes internally in order to support multiple encoding types

Feature Request

It would be great if OfflineIMAP could leverage the built-in email libraries to store messages as a byte array and avoid unnecessary conversions to strings. Currently, OfflineIMAP will read the input file as text and then does a hard conversion to bytes assuming utf-8 encoding when interfacing to the IMAP server. This results in exceptions when something other than plain ascii or utf-8 are present in the email (see bugs #43 and #44 ). Furthermore, an email can contain multiple encodings (see attached test mail) and thus a solution that searches for one form of encoding would be problematic.

The built in library can help keep track of the multiple encodings if needed and do conversions based on the capabilities of the IMAP server. I imagine it should even simplify the \r\n and \n conversions sprinkled through the code.

I have attached a test email and simple python script (both with txt extensions) that I used as a simple proof of concept.

Test Message containing several different encodings (mbox format for easy viewing in a mail client)

testmail-mime.txt

Simple Python3 script that attempts to keep the message intact while still adding the X-OfflineIMAP header

py3.email-copy.txt

Problem with imaplib2 and UTF-8?

Hi,

I debug offlineimap and I see that is not using utf-8 in the server communication. If the server supports utf-8, it is included in the capability reply as UTF8=ACCEPT:

imap.gmail.com writer:
 [imap]:   50:46.81 imap.gmail.com writer > b'CADB5 CAPABILITY\r\n'
DEBUG:OfflineImap:[imap]:   50:46.81 imap.gmail.com writer > b'CADB5 CAPABILITY\r\n'
imap.gmail.com reader:
 [imap]:   50:46.97 imap.gmail.com reader poll => [(5, 1)]
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader poll => [(5, 1)]
 [imap]:   50:46.97 imap.gmail.com reader rcvd 236
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader rcvd 236
 [imap]:   50:46.97 imap.gmail.com reader < b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584\r\n'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader < b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584\r\n'
 [imap]:   50:46.97 imap.gmail.com reader < b'CADB5 OK Success\r\n'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader < b'CADB5 OK Success\r\n'
imap.gmail.com handler:
 [imap]:   50:46.97 imap.gmail.com handler _put_response(b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584')
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler _put_response(b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584')
 [imap]:   50:46.97 imap.gmail.com handler untagged_responses[CAPABILITY] 1 += ["b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS C"]
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler untagged_responses[CAPABILITY] 1 += ["b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS C"]
 [imap]:   50:46.97 imap.gmail.com handler _put_response(b'CADB5 OK Success')
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler _put_response(b'CADB5 OK Success')
 [imap]:   50:46.97 imap.gmail.com handler _request_pop(b'CADB5', ('OK', [b'Success'])) [0] = b'CADB5'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler _request_pop(b'CADB5', ('OK', [b'Success'])) [0] = b'CADB5'
 [imap]:   50:46.97 imap.gmail.com handler CAPABILITY:b'CADB5'.ready.set
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler CAPABILITY:b'CADB5'.ready.set
 [imap]:   50:46.97 imap.gmail.com handler state_change_free.set
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler state_change_free.set

But imapilb2 is not enabling utf-8 by default and offlineimap is not parsing this capability.

I created an small patch to enable it (typ and dat are not used yet):

diff --git a/offlineimap/imapserver.py b/offlineimap/imapserver.py
index 0812b3b..73a3f09 100644
--- a/offlineimap/imapserver.py
+++ b/offlineimap/imapserver.py
@@ -606,6 +606,10 @@ class IMAPServer:
                 s_dat = [x.decode('utf-8') for x in dat[-1].upper().split()]
                 imapobj.capabilities = tuple(s_dat)
 
+            if "ENABLE" in imapobj.capabilities and \
+                    "UTF8=ACCEPT" in imapobj.capabilities:
+                typ, dat = imapobj.enable("UTF8=ACCEPT")
+
             if self.delim is None:
                 listres = imapobj.list(self.reference, '""')[1]
                 if listres == [None] or listres is None:

And is possible test it:

 [imap]:   50:46.97 Account sync gmail [sync] ENABLE ('UTF8=ACCEPT',)
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail [sync] ENABLE ('UTF8=ACCEPT',)
 [imap]:   50:46.97 Account sync gmail state_change_pending.acquire
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail state_change_pending.acquire
 [imap]:   50:46.97 Account sync gmail _request_push(b'CADB6', ENABLE, {}) = b'CADB6'
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail _request_push(b'CADB6', ENABLE, {}) = b'CADB6'
 [imap]:   50:46.97 Account sync gmail data=b'CADB6 ENABLE UTF8=ACCEPT'
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail data=b'CADB6 ENABLE UTF8=ACCEPT'
 [imap]:   50:46.97 Account sync gmail ENABLE:b'CADB6'.ready.wait
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail ENABLE:b'CADB6'.ready.wait
imap.gmail.com writer:
 [imap]:   50:46.97 imap.gmail.com writer > b'CADB6 ENABLE UTF8=ACCEPT\r\n'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com writer > b'CADB6 ENABLE UTF8=ACCEPT\r\n'
imap.gmail.com reader:
 [imap]:   50:47.03 imap.gmail.com reader poll => [(5, 1)]
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader poll => [(5, 1)]
 [imap]:   50:47.03 imap.gmail.com reader rcvd 41
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader rcvd 41
 [imap]:   50:47.03 imap.gmail.com reader < b'* ENABLED UTF8=ACCEPT\r\n'
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader < b'* ENABLED UTF8=ACCEPT\r\n'
 [imap]:   50:47.03 imap.gmail.com reader < b'CADB6 OK Success\r\n'
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader < b'CADB6 OK Success\r\n'
imap.gmail.com handler:
 [imap]:   50:47.03 imap.gmail.com handler _put_response(b'* ENABLED UTF8=ACCEPT')
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler _put_response(b'* ENABLED UTF8=ACCEPT')
 [imap]:   50:47.03 imap.gmail.com handler untagged_responses[ENABLED] 0 += ["b'UTF8=ACCEPT'"]
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler untagged_responses[ENABLED] 0 += ["b'UTF8=ACCEPT'"]
 [imap]:   50:47.03 imap.gmail.com handler _put_response(b'CADB6 OK Success')
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler _put_response(b'CADB6 OK Success')
 [imap]:   50:47.03 imap.gmail.com handler _request_pop(b'CADB6', ('OK', [b'Success'])) [0] = b'CADB6'
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler _request_pop(b'CADB6', ('OK', [b'Success'])) [0] = b'CADB6'
 [imap]:   50:47.03 imap.gmail.com handler ENABLE:b'CADB6'.ready.set
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler ENABLE:b'CADB6'.ready.set
 [imap]:   50:47.03 imap.gmail.com handler state_change_free.set
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler state_change_free.set

But, after this change, imaplib2 communication with the server is stuck in the line 1310 (self.state_change_pending.acquire()) with the first command (LIST):

Account sync gmail:
 [imap]:   50:47.03 Account sync gmail [async] LIST ('""', '""')
DEBUG:OfflineImap:[imap]:   50:47.03 Account sync gmail [async] LIST ('""', '""')
 [imap]:   50:47.03 Account sync gmail state_change_pending.acquire
DEBUG:OfflineImap:[imap]:   50:47.03 Account sync gmail state_change_pending.acquire
imap.gmail.com reader:
 [imap]:   51:17.06 imap.gmail.com reader poll => []
DEBUG:OfflineImap:[imap]:   51:17.06 imap.gmail.com reader poll => []
 [imap]:   51:47.09 imap.gmail.com reader poll => []
DEBUG:OfflineImap:[imap]:   51:47.09 imap.gmail.com reader poll => []
(repeat these 2 lines continuously)

Any help with this problem?
Thanks a lot.

PS: Full debug of these commands:

imap.gmail.com writer:
 [imap]:   50:46.81 imap.gmail.com writer > b'CADB5 CAPABILITY\r\n'
DEBUG:OfflineImap:[imap]:   50:46.81 imap.gmail.com writer > b'CADB5 CAPABILITY\r\n'
imap.gmail.com reader:
 [imap]:   50:46.97 imap.gmail.com reader poll => [(5, 1)]
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader poll => [(5, 1)]
 [imap]:   50:46.97 imap.gmail.com reader rcvd 236
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader rcvd 236
 [imap]:   50:46.97 imap.gmail.com reader < b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584\r\n'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader < b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584\r\n'
 [imap]:   50:46.97 imap.gmail.com reader < b'CADB5 OK Success\r\n'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com reader < b'CADB5 OK Success\r\n'
imap.gmail.com handler:
 [imap]:   50:46.97 imap.gmail.com handler _put_response(b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584')
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler _put_response(b'* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584')
 [imap]:   50:46.97 imap.gmail.com handler untagged_responses[CAPABILITY] 1 += ["b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS C"]
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler untagged_responses[CAPABILITY] 1 += ["b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS C"]
 [imap]:   50:46.97 imap.gmail.com handler _put_response(b'CADB5 OK Success')
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler _put_response(b'CADB5 OK Success')
 [imap]:   50:46.97 imap.gmail.com handler _request_pop(b'CADB5', ('OK', [b'Success'])) [0] = b'CADB5'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler _request_pop(b'CADB5', ('OK', [b'Success'])) [0] = b'CADB5'
 [imap]:   50:46.97 imap.gmail.com handler CAPABILITY:b'CADB5'.ready.set
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler CAPABILITY:b'CADB5'.ready.set
 [imap]:   50:46.97 imap.gmail.com handler state_change_free.set
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com handler state_change_free.set
Account sync gmail:
 [imap]:   50:46.97 Account sync gmail _get_untagged_response(CAPABILITY) => [b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS 
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail _get_untagged_response(CAPABILITY) => [b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS 
 [imap]:   50:46.97 Account sync gmail _untagged_response(OK, ?, CAPABILITY) => [b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS 
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail _untagged_response(OK, ?, CAPABILITY) => [b'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS 
 [imap]:   50:46.97 Account sync gmail [sync] ENABLE ('UTF8=ACCEPT',)
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail [sync] ENABLE ('UTF8=ACCEPT',)
 [imap]:   50:46.97 Account sync gmail state_change_pending.acquire
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail state_change_pending.acquire
 [imap]:   50:46.97 Account sync gmail _request_push(b'CADB6', ENABLE, {}) = b'CADB6'
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail _request_push(b'CADB6', ENABLE, {}) = b'CADB6'
 [imap]:   50:46.97 Account sync gmail data=b'CADB6 ENABLE UTF8=ACCEPT'
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail data=b'CADB6 ENABLE UTF8=ACCEPT'
 [imap]:   50:46.97 Account sync gmail ENABLE:b'CADB6'.ready.wait
DEBUG:OfflineImap:[imap]:   50:46.97 Account sync gmail ENABLE:b'CADB6'.ready.wait
imap.gmail.com writer:
 [imap]:   50:46.97 imap.gmail.com writer > b'CADB6 ENABLE UTF8=ACCEPT\r\n'
DEBUG:OfflineImap:[imap]:   50:46.97 imap.gmail.com writer > b'CADB6 ENABLE UTF8=ACCEPT\r\n'
imap.gmail.com reader:
 [imap]:   50:47.03 imap.gmail.com reader poll => [(5, 1)]
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader poll => [(5, 1)]
 [imap]:   50:47.03 imap.gmail.com reader rcvd 41
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader rcvd 41
 [imap]:   50:47.03 imap.gmail.com reader < b'* ENABLED UTF8=ACCEPT\r\n'
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader < b'* ENABLED UTF8=ACCEPT\r\n'
 [imap]:   50:47.03 imap.gmail.com reader < b'CADB6 OK Success\r\n'
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com reader < b'CADB6 OK Success\r\n'
imap.gmail.com handler:
 [imap]:   50:47.03 imap.gmail.com handler _put_response(b'* ENABLED UTF8=ACCEPT')
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler _put_response(b'* ENABLED UTF8=ACCEPT')
 [imap]:   50:47.03 imap.gmail.com handler untagged_responses[ENABLED] 0 += ["b'UTF8=ACCEPT'"]
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler untagged_responses[ENABLED] 0 += ["b'UTF8=ACCEPT'"]
 [imap]:   50:47.03 imap.gmail.com handler _put_response(b'CADB6 OK Success')
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler _put_response(b'CADB6 OK Success')
 [imap]:   50:47.03 imap.gmail.com handler _request_pop(b'CADB6', ('OK', [b'Success'])) [0] = b'CADB6'
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler _request_pop(b'CADB6', ('OK', [b'Success'])) [0] = b'CADB6'
 [imap]:   50:47.03 imap.gmail.com handler ENABLE:b'CADB6'.ready.set
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler ENABLE:b'CADB6'.ready.set
 [imap]:   50:47.03 imap.gmail.com handler state_change_free.set
DEBUG:OfflineImap:[imap]:   50:47.03 imap.gmail.com handler state_change_free.set
Account sync gmail:
 [imap]:   50:47.03 Account sync gmail [async] LIST ('""', '""')
DEBUG:OfflineImap:[imap]:   50:47.03 Account sync gmail [async] LIST ('""', '""')
 [imap]:   50:47.03 Account sync gmail state_change_pending.acquire
DEBUG:OfflineImap:[imap]:   50:47.03 Account sync gmail state_change_pending.acquire
imap.gmail.com reader:
 [imap]:   51:17.06 imap.gmail.com reader poll => []
DEBUG:OfflineImap:[imap]:   51:17.06 imap.gmail.com reader poll => []
 [imap]:   51:47.09 imap.gmail.com reader poll => []
DEBUG:OfflineImap:[imap]:   51:47.09 imap.gmail.com reader poll => []

Doing the same process manually, I dont have any problem:

tag3 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
tag3 OK Success
tag4 LIST "" ""
* LIST (\Noselect) "/" "/"
tag4 OK Success
tag5 ENABLE UTF8=ACCEPT
* ENABLED UTF8=ACCEPT
tag5 OK Success
tag6 LIST "" ""
* LIST (\Noselect) "/" "/"
tag6 OK Success

Any help?
Thanks
kix

How to install imaplib2?

General informations

system/distribution (with version): Void
offlineimap version (offlineimap -V): git master
Python version: 3.9
server name or domain: NA
CLI options: python3 offlineimap.py

Configuration file offlineimaprc

NA

Logs, error

Traceback (most recent call last):
  File "/home/c/data/programming/python/offlineimap3/offlineimap.py", line 19, in <module>
    from offlineimap import OfflineImap
  File "/home/c/data/programming/python/offlineimap3/offlineimap/__init__.py", line 20, in <module>
    from offlineimap.init import OfflineImap
  File "/home/c/data/programming/python/offlineimap3/offlineimap/init.py", line 34, in <module>
    from offlineimap import threadutil, accounts, folder, mbnames
  File "/home/c/data/programming/python/offlineimap3/offlineimap/accounts.py", line 26, in <module>
    from offlineimap.repository import Repository
  File "/home/c/data/programming/python/offlineimap3/offlineimap/repository/__init__.py", line 20, in <module>
    from offlineimap.repository.IMAP import IMAPRepository, MappedIMAPRepository
  File "/home/c/data/programming/python/offlineimap3/offlineimap/repository/IMAP.py", line 25, in <module>
    from offlineimap import folder, imaputil, imapserver, OfflineImapError
  File "/home/c/data/programming/python/offlineimap3/offlineimap/folder/__init__.py", line 4, in <module>
    from . import Base, Gmail, IMAP, Maildir, LocalStatus, UIDMaps
  File "/home/c/data/programming/python/offlineimap3/offlineimap/folder/Gmail.py", line 22, in <module>
    from offlineimap import imaputil, imaplibutil, OfflineImapError
  File "/home/c/data/programming/python/offlineimap3/offlineimap/imaplibutil.py", line 30, in <module>
    from imaplib2 import IMAP4, IMAP4_SSL, InternalDate
ImportError: cannot import name 'IMAP4' from 'imaplib2' (/home/c/.local/lib/python3.9/site-packages/imaplib2/__init__.py)

Steps to reproduce the error

I installed imaplib2 using pip pip install --user imaplib2
offlineimap3 compalined that could not import name IMAP4 from imaplib2.

foldersort broken: '<' not supported between instances of 'K' and 'K'

General information

system/distribution (with version): Debian unstable
offlineimap version (offlineimap -V): 7.3.0 (0.0~git20201110.74744ce+dfsg-1)
Python version: v3.9.0+

It seems that foldersort doesn't work if specified. If I set it to lambda x,y: -cmp(x,y) as per the example configuration file, offlineimap errors with

ERROR: While attempting to sync account 'madduck.net'
  '<' not supported between instances of 'K' and 'K'

Traceback:
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 725, in getfolders
    retval.sort(key=cmp2key(self.foldersort))

If I remove foldersort, it works.

IMAP4 protocol error: <class 'TypeError'>

General informations

system/distribution (with version): openSUSE Tumbleweed 20211029
offlineimap version (offlineimap -V): OfflineIMAP 7.3.4
Python version: python38-3.8.12-2.1.x86_64
server name or domain: imap.gmx.com
CLI options: none

Configuration file offlineimaprc

[general]
accounts = gmx

[Account gmx]
localrepository = gmx-local
remoterepository = gmx-remote

[Repository gmx-local]
type = Maildir
localfolders = ~/.gmx/Mail

[Repository gmx-remote]
type = IMAP
#added for troubleshooting, didn't work with "yes" either
starttls = no
#added for troubleshooting
ssl = yes
remotehost = imap.gmx.com
remoteport = 993
remoteuser = [email protected]
#not sure why offlineimap wouldn't use the system ca certs, but I had to add this manually
sslcacertfile = ~/.gmx/global-root-ca-chain-demos-digicert-com.pem
#added for troubleshooting, tried various options here, no difference
auth_mechanisms = GSSAPI, XOAUTH2, CRAM-MD5, PLAIN, LOGIN

Logs, error

$ offlineimap -d imap
OfflineIMAP 7.3.4
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.101 (bundled), Python v3.8.12, OpenSSL 1.1.1l  24 Aug 2021
Debug mode: Forcing to singlethreaded.
Now debugging for imap: IMAP protocol debugging
Now debugging for : Other offlineimap related sync messages
Account sync gmx:
 [imap]: Using authentication mechanisms ['GSSAPI', 'XOAUTH2', 'CRAM-MD5', 'PLAIN', 'LOGIN']
 *** Processing account gmx
 Establishing connection to imap.gmx.com:993 (gmx-remote)
 [imap]: gmx-remote: level 'tls_compat', version 'None'
 ERROR: While attempting to sync account 'gmx'
  IMAP4 protocol error: program error: <class 'TypeError'> - cannot use a bytes pattern on a string-like object
 ['  File "/usr/local/lib/python3.8/site-packages/offlineimap/accounts.py", line 293, in syncrunner\n    self.__sync()\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/accounts.py", line 372, in __sync\n    remoterepos.getfolders()\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/repository/IMAP.py", line 452, in getfolders\n    imapobj = self.imapserver.acquireconnection()\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/imapserver.py", line 557, in acquireconnection\n    imapobj = imaplibutil.WrappedIMAP4_SSL(\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/imaplibutil.py", line 202, in __init__\n    super(WrappedIMAP4_SSL, self).__init__(*args, **kwargs)\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/bundled_imaplib2.py", line 2191, in __init__\n    IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/bundled_imaplib2.py", line 399, in __init__\n    self.welcome = self._request_push(name=\'welcome\', tag=\'continuation\').get_response(\'IMAP4 protocol error: %s\')[1]\n', '  File "/usr/local/lib/python3.8/site-packages/offlineimap/bundled_imaplib2.py", line 200, in get_response\n    raise typ(exc_fmt % str(val))\n']
 *** Finished account 'gmx' in 0:00
 ERROR: Exceptions occurred during the run!
 ERROR: While attempting to sync account 'gmx'
  IMAP4 protocol error: program error: <class 'TypeError'> - cannot use a bytes pattern on a string-like object
 
Traceback:
  File "/usr/local/lib/python3.8/site-packages/offlineimap/accounts.py", line 293, in syncrunner
    self.__sync()
  File "/usr/local/lib/python3.8/site-packages/offlineimap/accounts.py", line 372, in __sync
    remoterepos.getfolders()
  File "/usr/local/lib/python3.8/site-packages/offlineimap/repository/IMAP.py", line 452, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/local/lib/python3.8/site-packages/offlineimap/imapserver.py", line 557, in acquireconnection
    imapobj = imaplibutil.WrappedIMAP4_SSL(
  File "/usr/local/lib/python3.8/site-packages/offlineimap/imaplibutil.py", line 202, in __init__
    super(WrappedIMAP4_SSL, self).__init__(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/offlineimap/bundled_imaplib2.py", line 2191, in __init__
    IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)
  File "/usr/local/lib/python3.8/site-packages/offlineimap/bundled_imaplib2.py", line 399, in __init__
    self.welcome = self._request_push(name='welcome', tag='continuation').get_response('IMAP4 protocol error: %s')[1]
  File "/usr/local/lib/python3.8/site-packages/offlineimap/bundled_imaplib2.py", line 200, in get_response
    raise typ(exc_fmt % str(val))

Additional notes

I currently have this account set up and working in claws. Setup was pretty straightforward. I've reviewed the configs, but I still may be overlooking something.
GMX IMAP info: https://support.gmx.com/pop-imap/imap/server.html

Errors on Windows: fcntl, signal.SIGHUP, OSError is not subscriptable, FileExistsError

Hi,
I run the last version of offlineimap3 (2021-01-05T08:55:15Z) on windows and had some issues.
Versions: Windows 10, Python 3.9.1 x64; imaplib2 3.06 (last from https://github.com/jazzband/imaplib2); rfc6555-0.0.0, portalocker-2.0.0, selectors2-2.0.2, distro-1.5.0 (from pypi).
offlineimap -V: offlineimap v7.3.0, imaplib2 v3.06, Python v3.9.1, OpenSSL 1.1.1g 21 Apr 2020

Run command: offlineimap -o -c test.conf -u Basic
Configuration file test.conf:

[general]
metadata = offlineimap.metadata
maxsyncaccounts = 1
ui = Basic
ignore-readonly = no
accounts = Test
[Account Test]
maildir-windows-compatible = yes
localrepository = Test_local
remoterepository = Test_remote
[Repository Test_local]
type = Maildir
localfolders = mail_Test
[Repository Test_remote]
sslcacertfile=ca-certificates.crt
type = IMAP
ssl = yes
remotehost = <imap_server>
remoteuser =
remotepass =
subscribedonly = no
readonly = True

Issues:

offlineimap\imaplibutil.py, line 18
ModuleNotFoundError: No module named 'fcntl'

To continue I just commented all lines with "fcntl".

offlineimap\init.py, line 75
info = "imaplib2 v%s, Python v%s" % (imaplib.__version__, PYTHON_VERSION)
AttributeError: module 'imaplib2' has no attribute __version__

To continue I replaced imaplib.__version__ with some string.

offlineimap\init.py
ERROR: module 'signal' has no attribute 'SIGHUP' ('SIGUSR1', 'SIGUSR2', 'SIGQUIT')

To continue I commented lines with "signal.SIGHUP" etc. Like here: https://github.com/hyperopt/hyperopt/pull/452/files/af8ecd2e14e4785517c8be080fa972e5bf539564

offlineimap\folder\GmailMaildir.py, line 186, offlineimap\folder\Maildir.py, lines 442, 528
TypeError: 'OSError' object is not subscriptable

I replaced "e[1]" with "e.errno".

offlineimap\folder\Base.py, line 284, and some more files
os.rename(uidfilename + ".tmp", uidfilename)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'offlineimap.metadata\Repository-Test_local\FolderValidity\Drafts.tmp' -> 'offlineimap.metadata\Repository-Test_local\FolderValidity\Drafts'

This error appear on second run. To work around the error I must remove "offlineimap.metadata" before run offlineimap3.

But in general, I have reached my goal - the letters are downloaded, thanks!

AttributeError: 'memoryview' object has no attribute 'decode'

General informations

system/distribution (with version): Debian sid
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.2, OpenSSL 1.1.1k 25 Mar 2021
Python version: 3.9.2
server name or domain: my own domain
CLI options: none (offlineimap)

Configuration file offlineimaprc

[general]
accounts = MyAccount

[Account MyAccount]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = ~/Maildir

[Repository Remote]
type = IMAP
remotehost = (mydomain)
remoteuser = (myuser)
remotepassfile = ...
ssl = yes
cert_fingerprint = ...
maxconnections = 1

Logs, error

Copy message from Remote:INBOX:
 ERROR: Copying message 30940 [acc: MyAccount]
  decoding with 'iso885"; 	name' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')
Thread 'Copy message from Remote:INBOX' terminated with exception:
Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/imaputil.py", line 406, in utf7m_decode
    for c in binary.decode():
AttributeError: 'memoryview' object has no attribute 'decode'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/threadutil.py", line 146, in run
    Thread.run(self)
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 802, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 342, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
  File "/usr/share/offlineimap3/offlineimap/folder/IMAP.py", line 908, in _fetch_from_imap
    ndata1 = self.parser['8bit-RFC'].parsebytes(data[0][1])
  File "/usr/lib/python3.9/email/parser.py", line 123, in parsebytes
    return self.parser.parsestr(text, headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 56, in parse
    feedparser.feed(data)
  File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
    self._call_parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 385, in _parsegen
    for retval in self._parsegen():
  File "/usr/lib/python3.9/email/feedparser.py", line 256, in _parsegen
    if self._cur.get_content_type() == 'message/delivery-status':
  File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
    return self[name](name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/lib/python3.9/email/headerregistry.py", line 445, in parse
    kwds['parse_tree'] = parse_tree = cls.value_parser(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2675, in parse_content_type_header
    ctype.append(parse_mime_parameters(value[1:]))
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2569, in parse_mime_parameters
    token, value = get_parameter(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2492, in get_parameter
    token, value = get_value(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 2403, in get_value
    token, value = get_quoted_string(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1294, in get_quoted_string
    token, value = get_bare_quoted_string(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1223, in get_bare_quoted_string
    token, value = get_encoded_word(value)
  File "/usr/lib/python3.9/email/_header_value_parser.py", line 1064, in get_encoded_word
    text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
  File "/usr/lib/python3.9/email/_encoded_words.py", line 181, in decode
    string = bstring.decode(charset)
AttributeError: decoding with 'iso885"; 	name' codec failed (AttributeError: 'memoryview' object has no attribute 'decode')

On the server, I found a file containing the UID 30940. Its content is UTF-8:

$ file cur/1374247664.30940_1.serv:2,S
./cur/1374247664.30940_1.serv:2,S: SMTP mail, UTF-8 Unicode text

And this mail contains headers:

Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"

Steps to reproduce the error

For me, just rerun offlineimap.

Please add 'offlineimap' tag and pid to syslog output

Currently offlineimap logs just the log message to syslog. That makes it difficult to identify the messages belonging to offlineimap.

Please consider to add an offlineimap tag and the pid of the running script to the syslog message (analogous to logger -t <tag> -i <message>).

Should oauth_xxxxx_eval functions return str or bytes?

General informations

system/distribution (with version): Arch Linux, kernel 5.11.1-arch1-1
offlineimap version (offlineimap -V):
- from offlineimap -V: offlineimap v7.3.0, imaplib2 v3.06, Python v3.9.2, OpenSSL 1.1.1j 16 Feb 2021
- it's actually offlineimap3 commit 1e7ef9e with patches from #55 and changes according to jazzband/imaplib2#15)
Python version: 3.9
server name or domain: gmail
CLI options: None

Configuration file offlineimaprc

[general]
metadata = $XDG_DATA_HOME/offlineimap
accounts = [email protected] 
pythonfile = $XDG_CONFIG_HOME/offlineimap/offlineimap_funcs.py

[mbnames]
enabled = yes
filename = $XDG_CONFIG_HOME/offlineimap/mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"
incremental = no

[Account [email protected]]
localrepository = [email protected]
remoterepository = [email protected]
maxage = 60
synclabels = yes

[Repository [email protected]]
type = GmailMaildir
localfolders = $HOME/Sync/MailDir/[email protected]
sep = /
sync_deletes = no
restoreatime = no

[Repository [email protected]]
type = Gmail
remoteuser = [email protected]
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
ssl_version = tls1_2
auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_gmail_client_id("[email protected]", "private-client.json")
oauth2_client_secret_eval = get_gmail_client_secret("[email protected]", "private-client.json")
oauth2_access_token_eval = get_gmail_token("[email protected]", "private-client.json")
oauth2_refresh_token_eval = get_gmail_refresh_token("[email protected]", "private-client.json")
expunge = yes
subscribedonly = no
createfolders = False
sync_deletes = yes
readonly = True

Python file `offlineimap_funcs.py`

See https://gist.github.com/piyueh/a2d65e095ea675a2c715ad42b7b61d10

Issue description

In the documentation in offlineimap.conf, it says the return values of the functions used by oauth2_client_id_eval, oauth2_client_secret_eval, oauth2_access_token_eval, and oauth2_refresh_token_eval should be bytes. However, returning bytes caused errors like this:

...
...
File "<redacted>/offlineimap/repository/IMAP.py", line 465, in getoauth2_client_id
  client_id = client_id.strip("\n")
TypeError: a bytes-like object is required, not 'str'

It is caused by that client_id is of bytes type, while the "\n" character is of str type. One possible fix is to use b"\n" instead. However, once I fixed the "\n" to b"\n" at line 431, 448, 465, and 482, a new error occurred:

*** Processing account [email protected]
Establishing connection to imap.gmail.com:993 ([email protected])
XOAUTH2 authentication failed: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'
ERROR: All authentication types failed:
        XOAUTH2: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'
*** Finished account '[email protected]' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: All authentication types failed:
        XOAUTH2: b'[AUTHENTICATIONFAILED] Invalid credentials (Failure)'

Traceback:
  File "<redacted>/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "<redacted>/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "<redacted>/offlineimap/repository/IMAP.py", line 672, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "<redacted>/offlineimap/imapserver.py", line 588, in acquireconnection
    self.__authn_helper(imapobj)
  File "<redacted>/offlineimap/imapserver.py", line 453, in __authn_helper
    raise OfflineImapError("All authentication types "

Then, I reverted back to using "\n" and return str instead of bytes from those evaluation functions. Now it's working!!!

So now I'm confused. It seems these evaluation functions (for oauth2_client_id_eval, oauth2_client_secret_eval, oauth2_access_token_eval, and oauth2_refresh_token_eval) should return str, not bytes. But the documentation in offlineimap.conf says they should return bytes. Can anyone help or shed some light? Thanks!!

Omitting sslcacertfile in favor of cert_fingerprint is not working

General informations

system/distribution (with version): Ubuntu 21.04
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1f 31 Mar 2020
Python version: Python 3.9.1+
server name or domain: See config file.
CLI options:

Configuration file offlineimaprc

This is a configuration to connect to Riseup.net email server over tor using their .onion address. They serve an HTTPs certificate that does not contain the onion address cname and thus validation fails without a matching domain name.

[Account riseup.net]
localrepository = riseup.net-local
remoterepository = riseup.net-remote
autorefresh = 1
quick = 20
socktimeout = 20

[Repository riseup.net-local]
type = Maildir
localfolders = <PATH>

[Repository riseup.net-remote]
type = IMAP
remotehost = 5gdvpfoh6kb2iqbizb37lzk2ddzrwa47m6rpdueg2m656fovmbhoptqd.onion
remoteport = 993
cert_fingerprint = af121b43c05056f1103461eb50d7ba45193a3f97
sslcacertfile =
ssl = yes
holdconnectionopen = yes
remoteuser = <USER>
remotepasseval = <PASSWORD>
idlefolders = ['INBOX']
folderfilter = lambda foldername: foldername in ['INBOX', 'Sent', 'Trash', 'Drafts']

Logs, error

ERROR: While attempting to sync account 'riseup.net' ('CA Cert verifying failed:  no matching domain name found in certificate',)

Steps to reproduce the error

Simply run: torsocks offlineimap -a riseup.net

Now I believe I found the problem. In repository/IMAP.py, the getsslcacertfile() is responsible to look for the sslcacertfile config value which if omitted is thus None and the fallback goes onto the CA system file path which is good. By default, it is good to always attempt to validate.

However, in this very specific case described in this ticket, the validation must be done through cert_fingerprint and thus omitting it or setting it to an empty string should be enough to ignore it but current code does not:

        conf_sslacertfile = self.getconf('sslcacertfile', None)
        if conf_sslacertfile == "OS-DEFAULT" or \
                conf_sslacertfile is None or \
                conf_sslacertfile == '':
            cacertfile = get_os_sslcertfile()

Both the empty string and the None value fallback to the CA system path and thus we never fallback to cert_fingerprint.

Either conf_sslacertfile == '' is removed or having a cert_fingerprint is enough to bypass this CA check if sslcacertfile is None or empty string? Or maybe a more explicit approach would be to make the user explicitly disable CA validation through an option... Unclear what is best from my perspective.

Note that the above used to work with offlineimap2 for which I have been using this for years.

offlineimap3: fails with "'bytes' object has no attribute 'encode'" error

@thekix First Debian bug for offlineimap3 - https://bugs.debian.org/973408.

And just fyi, Debian version is based on c850e74

offlineimap3 test trying to import offlineimap.virtual_imaplib2

test/OLItest/TestRunner.py is trying to import offlineimap.virtual_imaplib2 but offlineimap3 is not having that. And, so an initial Debian packaging attempt fails with: ModuleNotFoundError: No module named 'offlineimap.virtual_imaplib2'

Local IMAP server with "snake oil" cert gives error

General informations

system/distribution (with version): Debian bullseye
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i 8 Dec 2020
Python version: Python 3.9.1+
server name or domain: a local server
CLI options: no one (only 'offlineimap')

Configuration file offlineimaprc

[general]
metadata = ~/.var/offlineimap
accounts = Test
maxsyncaccounts = 1
ignore-readonly = no

[Account Test]
localrepository = Local-Test
remoterepository = Remote-Test
autorefresh = 10
[Repository Local-Test]
type = IMAP
remotehost =
remoteuser =
remotepass =
[Repository Remote-Test]
type = IMAP
remotehost =
remoteuser =
remotepass =
cert_fingerprint =

Logs, error

ERROR: Exceptions occurred during the run!
ERROR: Unknown SSL protocol connecting to host '' for repository 'Local-Test'. OpenSSL responded:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1123)

Steps to reproduce the error

(I think) use a local IMAP server with a self signed certificate

This works well with the old (python2 based) offlineimap

offlineimap3: 'int' object is not subscriptable when connecting to outlook.office365.com

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): 0.0~git20211018.e64c254+dfsg-1
Python version: 3
server name or domain: outlook.office365.com
CLI options:
Debian Bug: https://bugs.debian.org/997933

Configuration file offlineimaprc

[general]
accounts = one,two,three,...
#fsync false to conserve flash write cycles
fsync = false

[Account xxxxx]
localrepository = sLocal
remoterepository = sRemote

[Repository sRemote]
type = IMAP
ssl = yes
ssl_version = tls1
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
remoteprt = 993
remotehost = outlook.office365.com
remoteuser = [email protected]
folderfilter = lambda foldername: foldername in ['INBOX', 'Junk Email', 'Sent Items']

[Repository sLocal]
type = Maildir
localfolders = ~/Mail/xxxxx
sep = /
# Next again to help flash conservation & because noatime set
restoreatime = yes

pythonfile (if any)

Logs, error

$ offlineimap -a xxxxx
OfflineIMAP 8.0.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.9.7, OpenSSL 1.1.1l  24 Aug 2021
Account sync xxxxx:
 *** Processing account xxxxx
 Establishing connection to outlook.office365.com:993 (oRemote)
 ERROR: While attempting to sync account 'xxxxx'
  'int' object is not subscriptable
 *** Finished account 'xxxxx' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'xxxxx'
  'int' object is not subscriptable

Traceback:
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 681, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 683, in acquireconnection
    e.args[0][:35] == 'IMAP4 protocol error: socket error:':

Steps to reproduce the error

Try to fetch mail from outlook.office365.com

Two-factor authentication (2FA) in Symantec VIP Access (feature)

Can we please have offlineimap to sync our corporate email account that sits on outlook.office365.com, when its authentication asks for a two-factor authentication (2FA) that is a Symantec VIP Access one?

I guess we could open a valid session by authenticating in firefox, and then offlineimap could use the token that is stored in firefox cookie. Does it make sense? I guess this would need some upgrading in offlineimap, and we could re-do something from another python project that makes use of session tokens from firefox cookies. Do you remember any?

Does these code snippets help?

https://github.com/hile/jsontester/blob/99f71b0003954c1ca2cf0cb37a55f9398eac55b5/jsontester/cookies.py

https://stackoverflow.com/questions/49502254/how-to-import-firefox-cookies-to-python-requests

https://gist.github.com/denis-bz/88611952815b56b5b8b5

Thank you very much in advance for contributions on this challenge.

I have posted this on stack exchange as a question some months ago:
https://unix.stackexchange.com/q/630277/375983

Kind regards,

Bruno Schroeder

.travis.yml needs updating for offlineimap3

General informations

The .travis.yml is unchanged from the original offlineimap fork and is specifying python2. This should be updated to python3 and enabled on this repository.

offlineimap3 crashes with encoding error

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): 0.0~git20210225.1e7ef9e+dfsg-3
server name or domain: riseup.net
Debian bug: https://bugs.debian.org/986139

Logs, error

Last 11 debug messages logged for Copy message from Riseup-Remote:origin-mail-dir prior to exception:
thread: Register new thread 'Copy message from Riseup-Remote:origin-mail-dir' (account 'Riseup')
imap:   15:26.95 Copy message from Riseup-Remote:origin-mail-dir [async] UID ('FETCH', '1981', '(BODY.PEEK[])')
imap:   15:26.95 Copy message from Riseup-Remote:origin-mail-dir state_change_pending.acquire
imap:   15:26.95 Copy message from Riseup-Remote:origin-mail-dir state_change_pending.release
imap:   15:26.95 Copy message from Riseup-Remote:origin-mail-dir _request_push(b'EPBL24', UID, {}) = b'EPBL24'
imap:   15:26.95 Copy message from Riseup-Remote:origin-mail-dir data=b'EPBL24 UID FETCH 1981 (BODY.PEEK[])'
imap:   15:26.96 Copy message from Riseup-Remote:origin-mail-dir UID:b'EPBL24'.ready.wait
imap:   15:27.44 Copy message from Riseup-Remote:origin-mail-dir _get_untagged_response(FETCH) => [(b'184 (UID 1981 BODY[] {6664}', b'Return-Path: <[email protected]
imap:   15:27.44 Copy message from Riseup-Remote:origin-mail-dir _untagged_response(OK, ?, FETCH) => [(b'184 (UID 1981 BODY[] {6664}', b'Return-Path: <[email protected]
imap: Returned object from fetching 1981: 'Return-Path: <[email protected]>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on towhee.riseup.net
X-Spam-Level: 
X... v2.0
*****************************************
'
maildir: Write mail 'origin-mail-dir:1981' with flags set()
2021-03-30 10:15:27 INFO: OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
2021-03-30 10:15:27 WARNING: ERROR: Exceptions occurred during the run!
2021-03-30 10:15:27 WARNING: ERROR: Copying message 1981 [acc: Riseup]
  'ascii' codec can't encode characters in position 3523-3525: ordinal not in range(128)
2021-03-30 10:15:27 WARNING: 
Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 807, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 380, in savemessage
    tmpname = self.save_to_tmp_file(messagename, msg)
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 330, in save_to_tmp_file
    fd.write(msg.as_bytes(policy=output_policy))
  File "/usr/lib/python3.9/email/message.py", line 178, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.9/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.9/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.9/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.9/email/generator.py", line 268, in _handle_multipart
    self.write(subparts)
  File "/usr/lib/python3.9/email/generator.py", line 410, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))

Please do let me know if you need any additional information and I can ask the Debian user who reported the issue for the details.

We should use as default system sslcacertfile?

After the changes in the patch

fe25a5c

I think is possible to remove the sslcacertfile variable form the .offlineimaprc and use the system's default. This could be easy for new users. The current config is always like:

[Repository Remote-gmail]
type = Gmail
ssl = yes
remotehost = imap.gmail.com
remoteport = 993
remoteuser =user
remotepass = verysecurepass1
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

With the previouse patch, we can change the file to:

[Repository Remote-gmail]
type = Gmail
ssl = yes
remotehost = imap.gmail.com
remoteport = 993
remoteuser =user
remotepass = verysecurepass1
sslcacertfile = OS-DEFAULT

But my proposal is no variable, and then offlineimap uses OS-DEFAULT:

[Repository Remote-gmail]
type = Gmail
ssl = yes
remotehost = imap.gmail.com
remoteport = 993
remoteuser =user
remotepass = verysecurepass1

What do you think?

Regards,
kix

Error with Python 3.10 (and working in 3.9)

General informations

system/distribution (with version): docker with python:3-alpine image
offlineimap version (offlineimap -V): 8.0.0
Python version: 3.10

Logs, error

File "/usr/src/offlineimap3/offlineimap/imaplibutil.py", line 26, in
import rfc6555
File "/usr/local/lib/python3.10/site-packages/rfc6555.py", line 19, in
from selectors2 import DefaultSelector, EVENT_WRITE
File "/usr/local/lib/python3.10/site-packages/selectors2.py", line 25, in
from collections import namedtuple, Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/lib/python3.10/collections/init.py)

Steps to reproduce the error

build the following dockerfile and run the containter:

FROM python:3-alpine
WORKDIR /usr/src
RUN apk --no-cache add openssl wget
RUN pip3 install --no-cache-dir imaplib2 distro rfc6555 urllib3 certifi
RUN wget -nv https://github.com/OfflineIMAP/offlineimap3/archive/refs/heads/master.zip && unzip master.zip && rm master.zip && mv offlineimap3-master/ offlineimap3
WORKDIR /usr/src/offlineimap3
CMD [ "python3", "./offlineimap.py" ]

If I use the python:3.9-alpine image it works, but using the lastest python 3.10 does not work.
Searching the web for the error message I found this:
https://stackoverflow.com/questions/69381312/in-vs-code-importerror-cannot-import-name-mapping-from-collections
So it seems something changed from 3.9 to 3.10

[feature request] reduced output

Usecase : running offlineimap -o once or twice a day within a "do_the_internet.sh" script and be offline most of the time.

For that usecase, a reduced/summarized output would be appreciated that would display only the accounts synchronized and the folder where something was actually done (and with the number of message exchanged). If nothing, a simple "account X up-to-date, nothing to do".

Timestamp math causes uncaught exception

General informations

system/distribution (with version): Debian sid (up to date as of 2021-09-15, relevant details below)
offlineimap version (offlineimap -V): "7.3.0"
Python version: 3.9
server name or domain: fastmail.com
CLI options: none

Configuration file offlineimaprc

[general]
metadata = ~/.local/share/offlineimap
accounts = Fastmail

[Account Fastmail]
localrepository = FastmailLocal
remoterepository = FastmailRemote

[Repository FastmailLocal]
type = Maildir
localfolders = ~/Mail
#sep = .
sync_deletes = yes
utime_from_header = yes
filename_use_mail_timestamp = no


[Repository FastmailRemote]
type = IMAP
remotehost = imap.fastmail.com
remoteport = 993
remoteuser = [email protected]
#auth_mechanisms = GSSAPI, XOAUTH2, CRAM-MD5, PLAIN, LOGIN
remotepassfile = ~/.config/offlineimap/fastmail_passwd.txt
# TODO: Agent this!

cert_fingerprint = AF:01:8E:7F:FF:36:61:06:C6:F9:D4:38:D1:4E:08:98:78:C8:27:C5:E5:C0:A3:97:49:37:F2:76:90:B1:27:59
maxconnections = 3

pythonfile (if any)

nope

Logs, error

OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.9.7, OpenSSL 1.1.1l  24 Aug 2021
Account sync Fastmail:
 *** Processing account Fastmail
 Establishing connection to imap.fastmail.com:993 (FastmailRemote)
Folder Archive [acc: Fastmail]:
 Syncing Archive: IMAP -> Maildir
Folder Drafts [acc: Fastmail]:
 Syncing Drafts: IMAP -> Maildir
 Establishing connection to imap.fastmail.com:993 (FastmailRemote)
Folder fsme [acc: Fastmail]:
 Syncing fsme: IMAP -> Maildir
 Establishing connection to imap.fastmail.com:993 (FastmailRemote)
Folder INBOX [acc: Fastmail]:
 Syncing INBOX: IMAP -> Maildir
Folder INBOX/blocksfree [acc: Fastmail]:
 Syncing INBOX/blocksfree: IMAP -> Maildir
Folder INBOX/ghost1781 [acc: Fastmail]:
 Syncing INBOX/ghost1781: IMAP -> Maildir
Folder INBOX [acc: Fastmail]:
 Copy message UID 14901 (1/2) FastmailRemote:INBOX -> FastmailLocal:INBOX
 Copy message UID 14902 (2/2) FastmailRemote:INBOX -> FastmailLocal:INBOX
Folder INBOX/hunter [acc: Fastmail]:
 Syncing INBOX/hunter: IMAP -> Maildir
Folder INBOX/Gmail [acc: Fastmail]:
 Syncing INBOX/Gmail: IMAP -> Maildir
Copy message from FastmailRemote:INBOX:
 ERROR: Copying message 14901 [acc: Fastmail]
  offset must be a timedelta strictly between -timedelta(hours=24) and timedelta(hours=24), not datetime.timedelta(days=1, seconds=61200).
Thread 'Copy message from FastmailRemote:INBOX' terminated with exception:
Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 413, in savemessage
    date = self.get_message_date(msg, 'Date')
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 748, in get_message_date
    datetuple = parsedate_tz(msg.get(header))
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
    return self[name](name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/lib/python3.9/email/headerregistry.py", line 305, in parse
    value = utils.parsedate_to_datetime(value)
  File "/usr/lib/python3.9/email/utils.py", line 202, in parsedate_to_datetime
    tzinfo=datetime.timezone(datetime.timedelta(seconds=tz)))
ValueError: offset must be a timedelta strictly between -timedelta(hours=24) and timedelta(hours=24), not datetime.timedelta(days=1, seconds=61200).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/threadutil.py", line 146, in run
    Thread.run(self)
  File "/usr/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 815, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 421, in savemessage
    datestr = self.get_message_date(msg)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 748, in get_message_date
    datetuple = parsedate_tz(msg.get(header))
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
    return self[name](name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/lib/python3.9/email/headerregistry.py", line 305, in parse
    value = utils.parsedate_to_datetime(value)
  File "/usr/lib/python3.9/email/utils.py", line 202, in parsedate_to_datetime
    tzinfo=datetime.timezone(datetime.timedelta(seconds=tz)))
ValueError: offset must be a timedelta strictly between -timedelta(hours=24) and timedelta(hours=24), not datetime.timedelta(days=1, seconds=61200).


Last 2 debug messages logged for Copy message from FastmailRemote:INBOX prior to exception:
thread: Register new thread 'Copy message from FastmailRemote:INBOX' (account 'Fastmail')
maildir: Write mail 'INBOX:14901' with flags set()
ERROR: Exceptions occurred during the run!
ERROR: Copying message 14901 [acc: Fastmail]
  offset must be a timedelta strictly between -timedelta(hours=24) and timedelta(hours=24), not datetime.timedelta(days=1, seconds=61200).

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 815, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 421, in savemessage
    datestr = self.get_message_date(msg)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 748, in get_message_date
    datetuple = parsedate_tz(msg.get(header))
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 601, in __call__
    return self[name](name, value)
  File "/usr/lib/python3.9/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/lib/python3.9/email/headerregistry.py", line 305, in parse
    value = utils.parsedate_to_datetime(value)
  File "/usr/lib/python3.9/email/utils.py", line 202, in parsedate_to_datetime
    tzinfo=datetime.timezone(datetime.timedelta(seconds=tz)))

Steps to reproduce the error

1. Run offlineimap
1. There is no 2.

Debian details

Debian package version: 0.0~git20210825.4ca9c75+dfsg-1

-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-8-amd64 (SMP w/16 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
[Stupid proprietary wifi…]
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages offlineimap3 depends on:
ii ca-certificates 20210119
ii python3 3.9.2-3
ii python3-distro 1.6.0-2
ii python3-imaplib2 2.57-5.2

offlineimap3 recommends no packages.

Versions of packages offlineimap3 suggests:
ii python3-gssapi 1.6.12-1

-- no debconf information

Gmail FETCH error after updating offlineimap

General informations

system/distribution (with version): Debian GNU/Linux bullseye (20210217 updated)
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i 8 Dec 2020
Python version: Python 3.8.3
server name or domain: imap.gmail.com
CLI options: none

Configuration file offlineimaprc

[general]
accounts = GmailWork

[Account GmailWork]
localrepository = Gmaillocalmine
remoterepository = Gmailservermine
synclabels = yes
# This header is where labels go.  Usually you will be fine
# with default value (X-Keywords), but in case you want it
# different, here we go:
labelsheader = X-Keywords
postsynchook = notmuch new

[Repository Gmaillocalmine]
#This is the 'local' repository
type = GmailMaildir
localfolders = ~/Maildir/work2
sync_deletes = no

[Repository Gmailservermine]
type = Gmail
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
remoteuser = [email protected]

pythonfile (if any)

none

Logs, error

Folder 00order_mail2019 [acc: GmailWork]:
 Syncing 00order_mail2019: Gmail -> GmailMaildir
Folder 00order_mail2018 [acc: GmailWork]:
 ERROR: ERROR in syncfolder for GmailWork folder 00order_mail2018: Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 653, in syncfolder
    remotefolder.cachemessagelist()
  File "/usr/share/offlineimap3/offlineimap/folder/Gmail.py", line 132, in cachemessagelist
    res_type, response = imapobj.fetch("'%s'" % msgsToFetch,
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 807, in fetch
    return self._simple_command(name, message_set, message_parts, **kw)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/usr/lib/python3/dist-packages/imaplib2.py", line 1431, in _command_complete
    raise self.error('%s command error: %s %s. Data: %.100s' % (rqb.name, typ, dat, rqb.data))
imaplib2.IMAP4.error: FETCH command error: BAD [b'Could not parse command']. Data: b"MBKL15 FETCH '1:*' (FLAGS X-GM-LABELS UID)\r\n"

  FETCH command error: BAD [b'Could not parse command']. Data: b"MBKL15 FETCH '1:*' (FLAGS X-GM-LABELS UID)\r\n"

Steps to reproduce the error

updated offlineimap3 today
run offlineimap
enter password for Gmail
error is reproducible in another user in the same machine and another machine with same specs. In this case, it creates the local folders for downloading mail and then creates the error

Comment about UID 0 in search

Hi,

I was checking this code, and I found a strange situation. The server returns [b'1 2 3...'], then we split it and we get ['1', '2', '3',...]. This value is stored in res_data

            if ' ' in res_data[0] or res_data[0] == '':
                res_data = res_data[0].split()

Then, we check if the UID 0 is included:

            # Some servers are broken.
            if 0 in res_data:
                self.ui.warn("server returned UID with 0; ignoring.")
                res_data.remove(0)

We are checking an int in a str lists. So this code is not doing anything. It should be:

            # Some servers are broken.
            if '0' in res_data:
                self.ui.warn("server returned UID with 0; ignoring.")
                res_data.remove('0')

How is in Python 2? Probably we have the same issue.

To check this code, the easiest way is change this:

        if maxsize is not None:
            conditions.append("SMALLER %d" % maxsize)

        - if len(conditions) >= 1:
        + if len(conditions) >= 0:
            # Build SEARCH command.
            search_cond = "(%s)" % ' '.join(conditions)
            search_result = search(search_cond)

Regards,
kix

crash when uploading locally created no-utf-8 encoded message

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): 0.0~git20210105.00d395b+dfsg-2
Debian bug: https://bugs.debian.org/981485

Logs, error

OfflineIMAP 7.3.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i  8 Dec 2020
Account sync foo:
 *** Processing account foo
 Establishing connection to mail.foo.com:993 (fooRemote)
Folder INBOX [acc: foo]:
 Syncing INBOX: IMAP -> Maildir
 Copy message UID -2 (1/2) fooLocal:INBOX -> fooRemote:INBOX
 ERROR: Copying message -2 [acc: foo]
  'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
 ERROR: while syncing INBOX [account foo]
  'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
 ERROR: ERROR in syncfolder for foo folder INBOX: Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 666, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1186, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 1013, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 902, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in getmessage
    retval = file.read()
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte

  'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte
Account sync foo:
 *** Finished account 'foo' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: Copying message -2 [acc: foo]
  'utf-8' codec can't decode byte 0xfc in position 317: invalid start byte

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 902, in copymessageto
    message = self.getmessage(uid)
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 262, in getmessage
    retval = file.read()
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)

Steps to reproduce the error

Details at the Debian bug report. But a summary is:

Compose a message in mutt containing German umlauts. 
system uses utf-8 encoding, and mutt has a send_charset
option that defaults to `us-ascii:iso-8859-1:utf-8` and thus the
message was encoded as "Content-Type: text/plain; charset=iso-8859-1".
the error is seen while offlineimap tries to upload the sent message to IMAP server

Question: Delete message. Comparison with offlineipam (py2)

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): offlineimap3, latest commit in repo
Python version: 3.8.6
server name or domain: -
CLI options: Running offlineimap3.py with -1 -o

I want to check if this behavior is right:

Using mutt, enter in INBOX folder, mark one message for deletion, then delete it. Message is moved to Trash.
Run offlineimap3.py -1 -o and I get this output (snip)

INFO:OfflineImap:Syncing Trash: IMAP -> Maildir
Copy message UID -1 (1/1) Local-mymail:Trash -> Remote-mymail:Trash
INFO:OfflineImap:Copy message UID -1 (1/1) Local-mymail:Trash -> Remote-mymail:Trash
savemessage: Searching mails for new Message-ID failed. Could not determine new UID on Trash.
WARNING:OfflineImap:savemessage: Searching mails for new Message-ID failed. Could not determine new UID on Trash.

I run again offlineimap3.py -1 -o and I get:

INFO:OfflineImap:Syncing Trash: IMAP -> Maildir
Copy message UID 1125 (1/1) Remote-mymail:Trash -> Local-mymail:Trash
INFO:OfflineImap:Copy message UID 1125 (1/1) Remote-mymail:Trash -> Local-mymail:Trash

Running offlineimap3.py again, all is fine (nothing more about this mail). Mail is synced in local and in the server.

Is this behavior right? We are doing the sync in two steps.

Cheers,
kix

self signed certificate fails to verify

General informations

system/distribution (with version): Debian
offlineimap version (offlineimap -V): offlineimap3/0.0~git20210825.4ca9c75+dfsg-1
Python version:
server name or domain: kitenet.net
Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995403

Logs, error

ERROR: Unknown SSL protocol connecting to host 'kitenet.net' for repository 'kite'. OpenSSL responded:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1129)

Traceback:
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 686, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 661, in acquireconnection
    raise OfflineImapError(reason, severity, exc_info()[2])

Steps to reproduce the error

Use a mail server which has self signed certificate

Notes

Previously due to another bug report I had a4863b2 reverted in Debian. But I removed the revert when #66 was merged. But I did not test it with a self signed certificate. :(

traceback when using "remotepassfile" option

General informations

system/distribution (with version): Debian unstable
offlineimap version (offlineimap -V): offlineimap v7.3.0, imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i 8 Dec 2020
offlineimap Debian version: 0.0~git20210105.00d395b+dfsg-2
Debian bug: https://bugs.debian.org/981063

Configuration file offlineimaprc

Use remotepassfile in the offlineimaprc.

Logs, error

$ offlineimap -o -f test
OfflineIMAP 7.3.0
Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.05, Python v3.9.1+, OpenSSL 1.1.1i 8 Dec 2020
Account sync Gmail:
*** Processing account Gmail
Establishing connection to imap.gmail.com:993 (Gmail-Remote)
ERROR: While attempting to sync account 'Gmail'
sequence item 2: expected str instance, bytes found
*** Finished account 'Gmail' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'Gmail'
sequence item 2: expected str instance, bytes found

Traceback:
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 298, in syncrunner
self.__sync()
File "/usr/share/offlineimap3/offlineimap/accounts.py", line 374, in __sync
remoterepos.getfolders()
File "/usr/share/offlineimap3/offlineimap/repository/IMAP.py", line 648, in getfolders
imapobj = self.imapserver.acquireconnection()
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 592, in acquireconnection
self.__authn_helper(imapobj)
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 449, in __authn_helper
if func(imapobj):
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 375, in __authn_plain
imapobj.authenticate('PLAIN', self.__plainhandler)
File "/usr/lib/python3/dist-packages/imaplib2.py", line 691, in authenticate
typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
File "/usr/lib/python3/dist-packages/imaplib2.py", line 1684, in _simple_command
return self._command_complete(self._command(name, *args), kw)
File "/usr/lib/python3/dist-packages/imaplib2.py", line 1404, in _command
literal = literator(data, rqb)
File "/usr/lib/python3/dist-packages/imaplib2.py", line 2247, in process
ret = self.mech(self.decode(data))
File "/usr/share/offlineimap3/offlineimap/imapserver.py", line 217, in __plainhandler
retval = NULL.join((authz, authc, passwd))

Steps to reproduce the error

Use remotepassfile in the offlineimaprc.

a bytes-like object is required, not 'str'

General informations

system/distribution (with version): Fedora 34
offlineimap version (offlineimap -V): offlineimap v8.0.0, imaplib2 v3.06, Python v3.9.6, OpenSSL 1.1.1l 24 Aug 2021
CLI options: none

OfflineIMAP was installed with Nix and home-manager.

Configuration file offlineimaprc

[general]
accounts = perso
metadata = /home/cassou/.local/share/offlineimap
pythonfile = /home/cassou/.config/offlineimap/get_settings.py

[Account perso]
localrepository = perso-local
remoterepository = perso-remote

[Repository perso-local]
localfolders = /home/cassou/Mail/Perso
type = Maildir

[Repository perso-remote]
folderfilter = lambda folder: folder in [ 'Sent', 'INBOX', 'Archive' ]
remotehost = ...
remotepasseval = get_pass("perso", ['pass-show-password.sh','...']).strip("\n")
remoteport = 993
remoteuser = ...
ssl = yes
sslcacertfile = /etc/ssl/certs/ca-bundle.crt
starttls = no
type = IMAP

pythonfile (if any)

get_settings.py:

import subprocess

def get_pass(service, cmd):
    return subprocess.check_output(cmd, )

Logs, error

OfflineIMAP 8.0.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.06, Python v3.9.6, OpenSSL 1.1.1l  24 Aug 2021
Account sync perso:
 *** Processing account perso
 Establishing connection to choca.pics:993 (perso-remote)
 ERROR: While attempting to sync account 'perso'
  a bytes-like object is required, not 'str'
 *** Finished account 'perso' in 0:00
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'perso'
  a bytes-like object is required, not 'str'

Traceback:
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/accounts.py", line 298, in syncrunner
    self.__sync()
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/accounts.py", line 374, in __sync
    remoterepos.getfolders()
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/repository/IMAP.py", line 681, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/imapserver.py", line 593, in acquireconnection
    self.__authn_helper(imapobj)
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/imapserver.py", line 445, in __authn_helper
    if func(imapobj):
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/imapserver.py", line 371, in __authn_plain
    imapobj.authenticate('PLAIN', self.__plainhandler)
  File "/nix/store/fxyzhj7r70dshfkh8r1lipfrd2nffhsg-python3.9-imaplib2-3.6/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 712, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mechanism.upper())
  File "/nix/store/fxyzhj7r70dshfkh8r1lipfrd2nffhsg-python3.9-imaplib2-3.6/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 1705, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/nix/store/fxyzhj7r70dshfkh8r1lipfrd2nffhsg-python3.9-imaplib2-3.6/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 1425, in _command
    literal = literator(data, rqb)
  File "/nix/store/fxyzhj7r70dshfkh8r1lipfrd2nffhsg-python3.9-imaplib2-3.6/lib/python3.9/site-packages/imaplib2/imaplib2.py", line 2268, in process
    ret = self.mech(self.decode(data))
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/imapserver.py", line 211, in __plainhandler
    passwd = self.__getpassword()
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/imapserver.py", line 179, in __getpassword
    self.password = self.repos.getpassword() or \
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/repository/IMAP.py", line 585, in getpassword
    l_pass = self.localeval.eval(passwd)
  File "/…/offlineimap/lib/python3.9/site-packages/offlineimap/localeval.py", line 45, in eval
    return eval(text, names)
  File "<string>", line 1, in <module>

Steps to reproduce the error

Just run offlineimap

offlineimap / offlineimap3 Goto Github PK

offlineimap3's Introduction

OfflineIMAP

Description

Why should I use OfflineIMAP?

Project status and future

License

Downloads

Feedbacks and contributions

The community

Requirements & dependencies

Documentation

Read documentation locally

offlineimap3's People

Contributors

Stargazers

Watchers

Forkers

offlineimap3's Issues

General informations

General informations

Logs, error

Steps to reproduce the error

General informations

Logs, error

General informations

Configuration file offlineimaprc

Logs, error

Steps to reproduce the error

Step 1: create a test maildir

Step 2: run offlineimap

Description

Temporary solution

General informations

General informations

Configuration file offlineimaprc

General informations

Configuration file offlineimaprc

General informations

Repro steps

General informations

Logs, error

Steps to reproduce the error

General informations

Configuration file offlineimaprc

pythonfile (if any)

Logs, error

Steps to reproduce the error

Other comments

General informations

Logs, error

Steps to reproduce the error

General informations

Configuration file offlineimaprc

pythonfile (if any)

General informations

Configuration file offlineimaprc

Logs, error

Steps to reproduce the error

General informations

Configuration file offlineimaprc

Logs, error

Steps to reproduce the error

Solution to the error

General informations

Configuration file offlineimaprc

Logs, error

Steps to reproduce the error

Solution to the error

General informations

Configuration file offlineimaprc

pythonfile (if any)

Logs, error

Steps to reproduce the error

Feature Request

Test Message containing several different encodings (mbox format for easy viewing in a mail client)

Simple Python3 script that attempts to keep the message intact while still adding the X-OfflineIMAP header

General informations

Configuration file offlineimaprc

Logs, error

Python file `offlineimap_funcs.py`