Unread emails not marked as read in Maildir about astroid HOT 37 CLOSED

With the merge of #95 there are Actions for all write operations, this should make it easier to do the flag-sync.

from astroid.

Do you mean that it still has the tag 'new' or that the unread flag is
not removed on the server? I usually remove the 'new' tag after having
done any tagging on new mail in the notmuch post-new hook or my poll
script.

Neither really is the responsibility of astroid, notmuch can synchronize
some of the tags to maildir flags [0]. These will normally show up on
the server side as well. There are some configuration flags for notmuch
to set that up.

[0] https://notmuchmail.org/special-tags/

Hugo Roy writes on January 3, 2016 16:33:

I've noticed that reading an email in astroid removes the notmuch 'unread' tag but does not seem to affect the status of the email as 'new' in maildir.

Is there something wrong with my notmuch setup or should this be handled by astroid?

---

Reply to this email directly or view it on GitHub:
https://github.com/gauteh/astroid/issues/31

from astroid.

No, I remove the notmuch 'new' tag with a post-new hook. What I am concerned with are emails which are tagged as "unread" and which are, well, unread.

In Maildir, such emails will be in the "new" directory.

When I open and read such emails, astroid will correctly remove the "unread" tag but this isn't synced with the Maildir "new" directory, so when I sync this with the server (with offlineimap) the read/unread status isn't changed.

As the link suggests, notmuch can take care of "Synchronization between tags and maildir flags". The page says:

The above tags are recognized by the lib, and synchronized with the corresponding maildir flags through the following lib function calls:

   notmuch_message_maildir_flags_to_tags()
   notmuch_message_tags_to_maildir_flags()

from astroid.

Hugo Roy writes on January 3, 2016 21:31:

No, I remove the notmuch 'new' tag is removed with a post-new hook. What I am concerned with are emails which are tagged as "unread" and which are, well, unread.

In Maildir, such emails will be in the "new" directory.

When I open and read such emails, astroid will correctly remove the "unread" tag but this isn't synced with the Maildir "new" directory, so when I sync this with the server (with offlineimap) the read/unread status isn't changed.

As the link suggests, notmuch can take care of "Synchronization between tags and maildir flags". The page says:

The above tags are recognized by the lib, and synchronized with the corresponding maildir flags through the following lib function calls:

   notmuch_message_maildir_flags_to_tags()
   notmuch_message_tags_to_maildir_flags()

Hm, I don't think this is something astroid should do. You can have
notmuch dirs that don't have the new/ cur/ tmp/ structure. I am sure
this has been discussed at notmuch, have you found anything there?

from astroid.

From IRC:

21:48 if you read email using astroid, and astroid changes tags directly using libnotmuch (and not the notmuch cli commands), astroid needs to call notmuch_message_tags_to_maildir_flags to sync the tag changes to flag chances

I already have this in notmuch-config:

[maildir]
synchronize_flags=true

from astroid.

Hugo Roy writes on January 3, 2016 22:35:

From IRC:

21:48 if you read email using astroid, and astroid changes tags directly using libnotmuch (and not the notmuch cli commands), astroid needs to call notmuch_message_tags_to_maildir_flags to sync the tag changes to flag chances

Hm, ok! That seems cumbersome to me, I would expect notmuch new to do
that. Needs to be fixed though!

from astroid.

from astroid.

Hugo Roy writes on January 4, 2016 10:33:

Yes, for me too which is why i wasn't sure where the issue was exactly.

I suppose this is a valid discussion to continue in #notmuch :-)

After some discussion there it seems that some way of doing this
automatically through libnotmuch would be preferable, but there are a
bunch of considerations and tedious coding on the notmuch side to be
done. Short answer, won't be done for a while anyway, so any code
modifying tags should call the sync_back function as well. Some
considerations:

• syncback is configurable, preferably it should match notmuch-config

• config.cc should read ~/.notmuch-config (or $NOTMUCH_CONFIG) and use
  it for a bunch of default values, including:
  x sync_back
  x database path
  x exclude tags
  x other?

  these could be configured for instanse like this:

  db.sync_back_tag_to_maildir_flags=[yes|no|notmuch]

  with "notmuch" being default, meaning it is read from notmuch config.

• config.cc should not read notmuch-config when testing or in debug
  mode.

• files may be renamed after this, meaning that a sent message might not
  have the name after the call.

from astroid.

After some discussion there it seems that some way of doing this
automatically through libnotmuch would be preferable,

Yes that's what I expected too which is why I was surprised it didn't work with astroid.

but there are a
bunch of considerations and tedious coding on the notmuch side to be
done. Short answer, won't be done for a while anyway, so any code
modifying tags should call the sync_back function as well.

OK. Well, I think this makes sense because it looks like quite a blocker for people like me using several devices and interfaces for email (currently at work, using webmail) and where at least the unread/read status needs to be in sync.

config.cc should read ~/.notmuch-config (or $NOTMUCH_CONFIG) and use
it for a bunch of default values

Yes, this would be most useful. Maybe also for name, primary email, etc.

db.sync_back_tag_to_maildir_flags=[yes|no|notmuch]

with "notmuch" being default, meaning it is read from notmuch config.

• config.cc should not read notmuch-config when testing or in debug
  mode.

it makes sense to me but I'm not a developer :)

• files may be renamed after this, meaning that a sent message might not
  have the name after the call.

This may be related to some issues that I've run into with notmuch while using mutt sometimes (getting errors complaining about no file or directory, etc.). But I can't help much here.

Anyway, thanks a lot for looking into this. It looks like I may switch to astroid completely on my personal laptop instead of just experimenting :)

from astroid.

Hugo Roy writes on January 4, 2016 17:09:

This may be related to some issues that I've run into with notmuch while using mutt sometimes (getting errors complaining about no file or directory, etc.). But I can't help much here.

Anyway, thanks a lot for looking into this. It looks like I may switch to astroid completely on my personal laptop instead of just experimenting :)

Cool, the fastest thing to do is to just have a yes|no option without
integration to the notmuch-config yet. Might have to go for that first
as an intermediate step.

from astroid.

↪ janvier 4, 2016 10:25 , Gaute Hope:

Cool, the fastest thing to do is to just have a yes|no option without
integration to the notmuch-config yet. Might have to go for that first
as an intermediate step.

If that means astroid's unread email will also be synced back through
maildir as unread in other devices, I'm all for easiest and fastest :)

from astroid.

Having an option to parse notmuch-config for unset config options would be really great. See e.g. #63
As I said there I don't think it's good to do that automatically by default, but one could ask the user.
But if you look at https://notmuchmail.org/pipermail/notmuch/2016/021617.html the notmuch-devs don't think relying on ~/.notmuch-config is great, because it's only for the CLI and has no specified format.

from astroid.

from astroid.

Gaute Hope writes on January 17, 2016 15:12:

Yeah, perhaps that is not such a bad idea. One option to read values
from ~/.notmuch-config (or one specfied!) into the config and overwrite
the current/default values.

I think there is a boost ini parser that could read the .notmuch-config.

from astroid.

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values
from ~/.notmuch-config (or one specfied!) into the config and overwrite
the current/default values.

I think it would be better the other way around, so the user can
overwrite settings from notmuch-config in astroid. Reading
notmuch-config should be labeled as fallback solution.

from astroid.

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values
from ~/.notmuch-config (or one specfied!) into the config and overwrite
the current/default values.

I think it would be better the other way around, so the user can
overwrite settings from notmuch-config in astroid. Reading
notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to
run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid
config. I think it could easily lead to some weird behaviour if we read
it every time. In that case we would need to specify one astroid config
and one notmuch config, since a user might have several notmuch setups
specified using environment variables or command line arguments to
notmuch.

Relevant values are:

• account information
• mail db
• synchronize tags
• muted tags
• ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library,
I think the best is to still keep some detachment from the config file.

from astroid.

Excerpts from Gaute Hope's message of Januar 22, 2016 12:28 :

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values
from ~/.notmuch-config (or one specfied!) into the config and overwrite
the current/default values.

I think it would be better the other way around, so the user can
overwrite settings from notmuch-config in astroid. Reading
notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to
run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid
config. I think it could easily lead to some weird behaviour if we read
it every time. In that case we would need to specify one astroid config
and one notmuch config, since a user might have several notmuch setups
specified using environment variables or command line arguments to
notmuch.

That's not what I've been thinking of:

• add a config like "astroid.notmuch.fallback_config", default is ""
• read that config file (if specified) and populate db, excluded_tags
  etc. but do not write it to astroid-config
• if the user set e.g. astroid.notmuch.excluded_tags use that instead of
  the value read from notmuch_config
• probably one could add a program_option to change to a different
  notmuch.fallback_config.

Don't know if that makes sense to you.

Relevant values are:

• account information
• mail db
• synchronize tags
• muted tags
• ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library,
I think the best is to still keep some detachment from the config file.

---

Reply to this email directly or view it on GitHub:
https://github.com/gauteh/astroid/issues/31#issuecomment-173889181

from astroid.

Ah, yeah. That works. But lets specify the path of the fallback config.

Den lørdag 23. januar 2016 skrev ff2000 [email protected] følgende:

Excerpts from Gaute Hope's message of Januar 22, 2016 12:28 :

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values
from ~/.notmuch-config (or one specfied!) into the config and overwrite
the current/default values.

I think it would be better the other way around, so the user can
overwrite settings from notmuch-config in astroid. Reading
notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to
run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid
config. I think it could easily lead to some weird behaviour if we read
it every time. In that case we would need to specify one astroid config
and one notmuch config, since a user might have several notmuch setups
specified using environment variables or command line arguments to
notmuch.

That's not what I've been thinking of:

• add a config like "astroid.notmuch.fallback_config", default is ""
• read that config file (if specified) and populate db, excluded_tags
  etc. but do not write it to astroid-config
• if the user set e.g. astroid.notmuch.excluded_tags use that instead of
  the value read from notmuch_config
• probably one could add a program_option to change to a different
  notmuch.fallback_config.

Don't know if that makes sense to you.

Relevant values are:

• account information
• mail db
• synchronize tags
• muted tags
• ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library,
I think the best is to still keep some detachment from the config file.

---

Reply to this email directly or view it on GitHub:
https://github.com/gauteh/astroid/issues/31#issuecomment-173889181

—
Reply to this email directly or view it on GitHub
https://github.com/gauteh/astroid/issues/31#issuecomment-174170788.

from astroid.

Excerpts from Gaute Hope's message of Januar 23, 2016 11:59 :

Ah, yeah. That works. But lets specify the path of the fallback config.

Specifying the path by default would also resolve #63 automagically ;)
So probably not that bad.

Though this could potentially kill some special setups, where the db set
in ~/.notmuch-config is not what the user wants to use in astroid. As that
is set in astroid config, it will be overwritten by the logic described
in my last post, so everything is fine, again. But if he changes now to a
custom notmuch.[fallback_]config but forgets to unset notmuch.db he will
probably get unexpected results...
Also we probably want to completely ignore notmuch.*-config options if
the user specifies --notmuch-config cmdline-arg (if we implement it).

Another option would be deprecation of notmuch.* config options. Then we
can read them + warn that "this will be removed in Astroid 0.7, please
use astroid.notmuch.config" or similar.
If the user really wants some very special settings just for astroid, he
can copy ~/.notmuch-config, apply changes and specify that file in
"astroid.notmuch.config".

(And sorry for thinking out loud - there are many ways to deal with
this, and to break users setup ;))

Den lørdag 23. januar 2016 skrev ff2000 [email protected] følgende:

Excerpts from Gaute Hope's message of Januar 22, 2016 12:28 :

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values
from ~/.notmuch-config (or one specfied!) into the config and overwrite
the current/default values.

I think it would be better the other way around, so the user can
overwrite settings from notmuch-config in astroid. Reading
notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to
run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid
config. I think it could easily lead to some weird behaviour if we read
it every time. In that case we would need to specify one astroid config
and one notmuch config, since a user might have several notmuch setups
specified using environment variables or command line arguments to
notmuch.

That's not what I've been thinking of:

• add a config like "astroid.notmuch.fallback_config", default is ""
• read that config file (if specified) and populate db, excluded_tags
  etc. but do not write it to astroid-config
• if the user set e.g. astroid.notmuch.excluded_tags use that instead of
  the value read from notmuch_config
• probably one could add a program_option to change to a different
  notmuch.fallback_config.

Don't know if that makes sense to you.

Relevant values are:

• account information
• mail db
• synchronize tags
• muted tags
• ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library,
I think the best is to still keep some detachment from the config file.

---

Reply to this email directly or view it on GitHub:
https://github.com/gauteh/astroid/issues/31#issuecomment-173889181

—
Reply to this email directly or view it on GitHub
https://github.com/gauteh/astroid/issues/31#issuecomment-174170788.

---

Reply to this email directly or view it on GitHub:
https://github.com/gauteh/astroid/issues/31#issuecomment-174171545

from astroid.

I just started to refactor config access:

• make Astroid::config a private member
• add ptree Config::notmuch_config which is populated from ~/.notmuch_config
• create function "Astroid::config(string path)" to return specific
  ptree object:
  • config.notmuch_config if path == astroid.notmuch
  • else config.get_child (path)

That way we should be able to access the config as usual (except direct
"astroid.notmuch.db" which we can avoid or catch in Astroid::config()),
avoid writing notmuch-config values into astroid-config (seperate
ptree-object) - and restrict access to Astroid::config-object.

If you agree with the approach I will continue later. (Need some break,
think I'm falling ill...)

from astroid.

Looks good, afk this weekend so, no rush on my part.

Den lørdag 23. januar 2016 skrev ff2000 [email protected] følgende:

I just started to refactor config access:

• make Astroid::config a private member
• add ptree Config::notmuch_config which is populated from
  ~/.notmuch_config
• create function "Astroid::config(string path)" to return specific
  ptree object:
• config.notmuch_config if path == astroid.notmuch
• else config.get_child (path)

That way we should be able to access the config as usual (except direct
"astroid.notmuch.db" which we can avoid or catch in Astroid::config()),
avoid writing notmuch-config values into astroid-config (seperate
ptree-object) - and restrict access to Astroid::config-object.

If you agree with the approach I will continue later. (Need some break,
think I'm falling ill...)

—
Reply to this email directly or view it on GitHub
https://github.com/gauteh/astroid/issues/31#issuecomment-174189143.

from astroid.

@hugoroy you might want to test #74
Everything works here, tests pass. If anything fails please comment in the pull request.

Note: That branch reads the notmuch config file. It defaults to ~/.notmuch-config. So make sure maildir.synchronize_flags is set to true there.

from astroid.

Tested for unread/read mail, it works.

from astroid.

↪ janvier 24, 2016 2:32 , Aurélien Ooms:

Tested for unread/read mail, it works.

This works for me, including for flaggeed status as well :)

from astroid.

Aurélien, Hugo: Thx for testing.
So everything works, and you didn't have any weird issues? Great :)
Notmuch will automatically sync tags to flags for these cases:

• 'D' iff the message has the "draft" tag
• 'F' iff the message has the "flagged" tag
• 'P' iff the message has the "passed" tag
• 'R' iff the message has the "replied" tag
• 'S' iff the message does not have the "unread" tag

(copied from /usr/include/notmuch.h)

@gauteh There are two more changes I plan:

• Performance: first iterate over messages, then over tags. That way I
  do not have to sync one message multiple times, just because several
  tags changed.
• Safety: Freeze message while tagging (notmuch_message_freeze)

So I think a seperate PR makes sense.

from astroid.

ff2000 writes on January 25, 2016 8:32:

Aurélien, Hugo: Thx for testing.
So everything works, and you didn't have any weird issues? Great :)
Notmuch will automatically sync tags to flags for these cases:

• 'D' iff the message has the "draft" tag
• 'F' iff the message has the "flagged" tag
• 'P' iff the message has the "passed" tag
• 'R' iff the message has the "replied" tag
• 'S' iff the message does not have the "unread" tag

(copied from /usr/include/notmuch.h)

@gauteh There are two more changes I plan:

• Performance: first iterate over messages, then over tags. That way I
  do not have to sync one message multiple times, just because several
  tags changed.
• Safety: Freeze message while tagging (notmuch_message_freeze)

So I think a seperate PR makes sense.

Excellent!

I think this should go in a separate PR, but you can base it off the
refactored-config PR.

Some issues:

• there are more places where I modify tags, these should perhaps be
  wrapped in Db:: calls, currently only threads (not messages) are
  changed in the Db class. grep for add_tag and remove_tag.

• the filename might change while the message is loaded somewhere, e.g.:

  1. open a thread
  2. go back to thread-index
  3. change a label
  4. go back to thrad-view
  5. try to save thread

  or:
  an unread thread is opened, the unread tag is removed, the message
  flags should be synced, now the filename is no longer the same.

So, some questions or approaches to that:
is the notmuch message and thread object still valid after sync has
been run? in that case we can make sure that filenames are never
cached, but always queried from the db.

if not, we might have to reload the thread/message on every sync call
that resulted in a change. perhaps this can be achieved through
a message_filename_changed (string messageid, string newfname) signal.

regards, gaute

from astroid.

By the way; ner had a fairly nice (but perhaps incomplete) c++ wrapper for the notmuch library: https://github.com/gauteh/astroid/pull/44#issuecomment-170337316

from astroid.

This was working but it seems that it was not merged in the last release

from astroid.

from astroid.

That would be awesome. This is 1 of the 3 reasons why I still need to open mutt nowadays :-)

(the other 2 being: sending encrypted email, and getting rids of some drafts that for some reason aren't tagged properly)

from astroid.

Hugo Roy writes on March 14, 2016 23:25:

That would be awesome. This is 1 of the 3 reasons why I still need to open mutt nowadays :-)

Some comments on the patch from @ff2000:

• You are calling the notmuch__synchronize_ function for each tag, it
  should only be called once for all tags.
• I think it fits better in on_message and on_thread? or in Actions.
  Grep the code and check if we add or remove tags anywhere else than
  the two functions you have now.

(the other 2 being: sending encrypted email, and getting rids of some drafts that for some reason aren't tagged properly)

2. Yes, that would be nice.

3. You can manually tag the message as a draft from the thread-view.

from astroid.

I've made myself a little workaround script that reset unread tag using notmuch cli tool before syncing with offlineimap, maybe it could be useful to anyone as I guess it can be used as astroids polling script: gist

from astroid.

Please test #112.

from astroid.

↪ avril 4, 2016 4:18 , Gaute Hope:

Please test #112.

Yes, the basics work (marking read/unread). I'll test for some time and
see if any issue comes up.

Thank you!

from astroid.

Fixed.

from astroid.

Thank you!

from astroid.

Thank you

from astroid.