Allow downloading filters from GMail. about gmailctl HOT 18 CLOSED

Totally. That's what I'm actually implementing in the remote-import branch. Alpha preview available soon.

from gmailctl.

If you (in a regular text editor, not using gmailctl) comment out most of the sample file, you can then run gmailctl diff and get a YAML file with your current configuration (with each line prefixed by - because it’s a diff to nothing). However, that file doesn’t seem to be in any of gmailctl’s input formats.

If we get import like this, it would also be nice if it detected new filters added on the server and also imported those even when you aren’t starting from scratch. I doubt it’s possible to import filters which were modified server-side (because running jsonnet backwards would be quite challenging), but at least reporting them and their differences would be nice.

Even if those are too difficult, though, an option in gmailctl init for “would you like to start from scratch, see an example file, or import your existing filters” would be much appreciated.

from gmailctl.

I agree that this is a nice to have. Here are my thoughts on that:

gmailctl already imports the remote filters to compute the diff with the local ones, but, as @dhouck correctly noticed, this format is not the input one, but something in between the local and the remote format (package gmailctl/pkg/filter). The real challenge is to be able to convert this one back to jsonnet, which is a higher level abstraction. I would have to somewhat decompile the filters to make this possible.

From easy to hard:

1. Add a dump_remote command that prints the remote filters in the intermediate format (without you having to do tricks like commenting the config file).
2. Convert the intermediate format into JSON (not Jsonnet).
3. Convert that into Jsonnet.
4. Correctly parse the filters into the right operands and functions available in the config.
5. Allow filters edit through the Gmail web interface and auto-import in the local config.

1 is basically there already, so no sweat. 2 needs some form of parsing, to get from bare strings to at least decomposed and, or, not, in order to print a minimally decent JSON. Since JSON is a subset of Jsonnet, adding an import option to gmailctl init becomes possible. To make 3 possible I need a Jsonnet pretty printer, which I don't have. I could create a minimal one, just to get rid of the extra quotes and add commas here and there. 4 is very hard. I would have to implement a correct parser for Gmail filters for all operands involved. 5. is impossible, since I would have to manipulate the Jsonnet AST, inject changes and pretty print it back (comments included). But more than the technical challenge there's a conceptual one. The config file would not be the source of truth anymore, making everything way more difficult and confusing.

TL;DR I can do 1 very easily, get to 3 in a few hours of work (I think), and I won't be able to get past that. This means that the imported filters won't be very pretty, so they will probably need a lot of refactoring work from you anyway.

Let me know if it still makes sense to do.

from gmailctl.

I was assuming that 4 was practically impossible, and certainly is if you want to do things like notice "There are a lot of pairs of filters, one with <query> that applies a label and one with <query> -to:<address> which skips the inbox; I should refactor that out"; I was expecting that imported filters would require manual refactoring.

For importing filters later, I was hoping to notice things like "The current config file compiles to this set of filters. The GMail settings have all of those filters, but also these other ones", and then you could import them as you do initially by just inserting them at the end of the Jsonnet file. Anything more complicated than that I was thinking it would be nice just to see a pre-diff for. If it's more complicated than that, seeing a diff if nonempty at the start of running gmailctl edit would still be useful to remind you that you did make change.

from gmailctl.

FWIW, I'd find it convenient to import my current filters, which are a horrible mess, but don't care about ongoing bi-directional sync or getting the output into a smart, high level form. I'd just find it vaguely convenient to have a basis to ensure I get all of them in my initial import, as far as possible.

from gmailctl.

The remote-import branch now contains a (pretty basic), but hopefully functioning implementation of the import command. Can someone please give it a try and give me some feedback?

Please note that the config file generated by the branch is not compatible with the stable version of gmailctl. If the current implementation is good enough I'll have to bump the config file version, handle the previous format transparently, update the docs, etc... So, please try this version but don't use it for real, since it's not stable!

from gmailctl.

I'll try it soon, probably later tonight.

from gmailctl.

I probably messed up with building (I don't know much about Go), but I can't run the import command.

If you want to help me debug my setup, here're some details:

$ git log -n 1
commit f590de2 (HEAD -> remote-import, origin/remote-import)
Author:     Michele Bertasi <[email protected]>
AuthorDate: Tue Apr 9 00:29:58 2019 +0000
Commit:     Michele Bertasi <[email protected]>
CommitDate: Tue Apr 9 00:29:58 2019 +0000

    Draft import command.

    The command now outputs a correct configuration file.
    Few parameters need to be added, plus documentation.

Error: unknown command "import" for "gmailctl"

Did you mean this?
        export

Run 'gmailctl --help' for usage.
unknown command "import" for "gmailctl"

Did you mean this?
        export

from gmailctl.

@dhouck maybe you are not inside the $GOPATH directory.

echo $GOPATH
mkdir -p $GOPATH/src/github.com/mbrt/gmailctl
cd $GOPATH/src/github.com/mbrt/gmailctl
rm -rf *
git clone https://github.com/mbrt/gmailctl.git .
git checkout remote-import
go build ./cmd/gmailctl

Then you should have the right binary there:

./gmailctl import

from gmailctl.

I was able to download and run import on my quite complicated gmail filter set! ./gmailctl diff shows no difference against the exported file!

A couple things I noted (none critical):

• I find the term import a bit confusing since that is what gmail calls uploading new filters. I would have guessed export since that is gmail calls downloading, but I see export is already taken (though maybe that could be revisted).
• Simple gmail filters like URGENT are represented as query: "URGENT rather than has: "URGENT".
• from:([email protected]) to:me is represented as:[{ from: "[email protected]"},{query: "to:me"}], but last part should probably be to: "me".
• What does isRaw signify? It seems to mean special characters should be ignored by gmailctl (though JSON escaping will obviously be applied when paring). If so, should query strings with special characters also be marked isRaw?
• And sort order is the order of the filters in gmail (which seems right)?

Thanks!
d

from gmailctl.

Thank you for the feedback!

1. Yes, I was struggling with the naming a you can see. The point of view is yours but reversed, considering the local file as source of truth, so than you export from it and you import to it. I'm open to suggestions though.
2. This is intended. The different between query and has is that with the latter, operands with spaces will be quoted in the resulting filter. If you happen to have a filter with spaces, or already quoted terms, has will incorrectly quote the terms again. query instead has no logic so it will happily upload the raw contents.
3. You're right I think, but it depends on how the filter actually is. If you edit it from the web interface you should see if the term to:me is present in the Has words: input, or if it was simply me in the To: section. The APIs make this distinction, even if there's no practical difference between the two.
4. This is again to avoid interpreting the string inside the operator. If in your config you have { from: 'foo bar' } the resulting filter in Gmail will be the literal "foo bar" (note the quotes). If this feature wasn't present you'd have to quote your strings in the config appropriately. The isRaw is an escape hatch for that. Again I could have used query, but Gmail is really picky and will see a difference between using the From: field or using the Has the words: fields.
5. Don't rely on the order of the filters. It changes as filter change. Write your config to be immune to that.

I hope this is somewhat clear. There's a lot of what it seems accidental complexity in Gmail filters, that is outside my control.

Perhaps I should think again about the isRaw modifier though as it's really ugly and not very clear. Maybe there's a way to extend the query operator in a retro-compatible way and make everyone happy. I'll think about it a little more.

from gmailctl.

For 1., you could call the command-line options upload and download, which seem clear and unambiguous.

from gmailctl.

I was able to get the remote-import branch to work; thanks for your help.

Two notes:

1. I see the following warning:

   Warning: Error getting one or more filters from Gmail: 1 error occurred:
           * error importing filter 'ANe1Bmh8i4SM8jaQyV5FO9nhEMwN2epwBBEk1w': error importing action: 
   empty action
   
   They will be ignored in the diff.
   

   This warning is completely accurate and correct; when I check with the "Try it now" option at https://developers.google.com/gmail/api/v1/reference/users/settings/filters/get, I see that it just checks if there's a certain string in the To: address, and has no actions. However, I would have liked to see what the rest of it was, to help me decide if I need to do anything about this warning.

2. One of the successfully imported filters reads:

       {
         filter: {
           to: <redacted>
         },
         actions: {}
       },
   

   Searching for the to: address in the output of https://developers.google.com/gmail/api/v1/reference/users/settings/filters/list, this is from a forwarding filter (see #56). There was no warning or error message for this; I only noticed it because of the empty action. Other filters I have forward as part of their actions, and that part was silently dropped. There should be a warning or error message if some part of the filter cannot be imported.

from gmailctl.

Thanks for the testing.

1. Noted in #53.
2. Makes sense, this should be addressed before merging.

from gmailctl.

+1 for upload and download. It avoid gmails terms, and I think they have the inherent "my machine to/from server" connotation that makes direction clearer.

from gmailctl.

(Given that you already have export I'd recommend keeping it as a synonym even if you do go with upload/download, to avoid breaking things)

from gmailctl.

download sounds good to me. However upload would be misleading. The export command actually doesn't upload anything. It takes the local configuration and produces an XML compatible with the Gmail import functionality, so I'd rather keep the existing name.

from gmailctl.

Alright, everything should be in a decent state now. I've merged this into the cfg_alpha3 branch. I'll do the config version bump and implement the migration. I might still change the config format in the meantime, and bundle other config changes before merging everything to master.

from gmailctl.