jmapio / jmap-perl Goto Github PK
View Code? Open in Web Editor NEWJMAP Proxy implemented in Perl
License: MIT License
JMAP Proxy implemented in Perl
License: MIT License
Instead, create an intermediate state with JUST the current changes, then sync later and bring things fully into sync. This means JMAP clients may see things out of order, but they're still going to be valid state transitions that bring you to the same point, and they mean that setFoos won't be as expensive.
The README says:
NOTE: this library does not ship with the cloned FastMail web interface which
is running on https://proxy.jmap.io/. The licensing situation for that code is
not yet resolved.
Is that what https://github.com/jmapio/jmap-demo-webmail is? It's README says:
There's a hosted version at http://proxy.jmap.io, which uses the JMAP proxy to provide a JMAP
interface to a Gmail or IMAP account.
I accidentally sent a bogus setMailboxes request and surprisingly got a completely bizarre response back:
~/code/topicbox$ ./jc setMailboxes '{create:{foo:{}}}'
[
[
"mailboxesSet",
{
"accountId" : "11049eb6-1dcc-11e6-82c1-f23c91556942",
"created" : {
"foo" : {
"id" : null
}
},
"destroyed" : [],
"newState" : null,
"notCreated" : {},
"notDestroyed" : {},
"notUpdated" : {},
"oldState" : null,
"updated" : []
},
"a"
]
]
I'd expect some kind of error, not a "I created it, and its id is null
".
Not only can we split locking so that calendar / contact fetches don't lock mail and vice versa, but we can track deletedModseq separately so we don't invalidate as much on clients.
Trying to use the live version of the proxy (https://proxy.jmap.io/) gives a "Sign in with Google temporarily disabled for this app" error for Gmail. Using an IMAP connection gives a "Account {id} does not exist, please re-create" error.
Trying to run the project locally fails during the install instructions with an error about the libemail-address-xs-perl
package. Changing this to libemail-address-perl
(which appears to provide the same packages, but could be wrong) gives the following much later in the build:
Use of uninitialized value $Net::CardDAVTalk::VERSION in concatenation (.) or string at t/00-load.t line 13.
(Tested on both Debian Jessie and Ubuntu 16.04.)
Am I doing something wrong here? Was hoping to be able to test out JMAP!
According to the spec, both "from" and "to" are lists of objects:
from: Emailer[]|null An array of name/email objects (see below) representing the parsed From header of the email, in the same order as they appear in the header. If the email doesn’t have a From header, this is null. If the header exists but does not have any content, the response is an array of zero length.
to: Emailer[]|null An array of name/email objects (see below) representing the parsed To header of the email, in the same order as they appear in the header. If the email doesn’t have a To header, this is null. If the header exists but does not have any content, the response is an array of zero length.
However, JMAP Proxy only accepts an object for "from" when you call setMessages:
([
[
setMessages => {
create => {
1 => {
mailboxIds => [8], # outbox
from => { email => "test\@example.com" },
to => [ { email => "test2\@example.com" } ],
subject => "Test mail",
textBody => "Hello there!",
},
}
},
],
]
@rjbs points out that the spec changed; at one point from used to be an object and not a list of objects:
I suspect JMAP Proxy was just never updated to account for this.
Thanks!
If a mailbox is deleted at the remote (IMAP) end, IMAP syncing will fail in several ways, and no new syncs will occur. Here's the report I had originally given via private email, here for easy future reference:
I noted this error:
ERROR Not a HASH reference at /home/jmap/jmap-perl/JMAP/ImapDB.pm line 672.
on c4de019e-1c5e-11e6-b873-f23c91556942:sync (dropping backend)
It's one place where the problem can occur. Here's that line, reformatted:
next if (
$status->{$row->{imapname}}{uidvalidity} == $row->{uidvalidity}
and $status->{$row->{imapname}}{highestmodseq}
and $status->{$row->{imapname}}{highestmodseq} == $row->{highestmodseq}
);
The undef being hash-dereferenced is $status->{$row->{imapname}}
. We got the status for every folder, but the status entry for a deleted folder is \'no'
and not a hash ref. (\$string)->{...}
is fatal.
I started by just adding another "next"-ing condition, but this comes up elsewehere, too. Many of the imap_*
methods in JMAP::Sync::Common rely on being able EXAMINE folders that may have been deleted.
I look at this page: https://proxy.jmap.io/
This is the first text I read:
The JMAP proxy is a work in a progress. It is currently stable enough to test out and get a feel for JMAP in action. All the methods in the spec are implemented, though some atomic guarantees are not possible with other users accessing your servers at the same time
Please start with an introduction. What is JMAP proxy all about? Then the details about the state (work in progress).
Related: https://github.com/guettli/programming-guidelines#love-your-docs
Does the proxy have one yet? I don't think so, but that's now a problem because I need to discover the download endpoint to use a message blobId.
It would be great to have better input/output validation in a separate spec file for typed data so the code is simpler and we're sure each field is tested against a schema - both in what we read, but most importantly in what we produce, so we're always talking valid JMAP to clients.
Step to reproduce:
If I stop and restart the proxy server (server.pl
):
Lots of things in ImapDB aren't really IMAP specfic, and mean it's not so easy to plug in a "purelocal" server version
It's because of the BINARY fetch command the proxy is sending.
I don't understand exactly why as I'm no IMAP expert, but using BODY instead works fine.
bin/server.pl needs MIME::Base64::URLSafe but it doesn't seem to get installed by the Makefile
For several reasons, but the main one is that the data gets stored in a TEXT column in the sqlite DB.
The sort criteria and filter criteria for messagelists are different for no good reason - they should have the same name.
Just a reminder to either me or @brong to get it up there sometime. Makes installing the proxy rather difficult :)
It should be fetching from a wrapper that allows us to sub out sqlite with another DB engine more easily
I found two issues:
More a feature request than a bug: It would be great if you could add a Vagrant file // Dockerfile to get the proxy up and running locally if people - like me - just want to test it out.
If you try to sign up with gmail through the proxy UI and you have no config file (or it's incomplete) the errors the server gives in the logs aren't clear.
The last update is Jul 15, 2018:
https://github.com/jmapio/jmap-perl/commits/master
Spec has changed quite a bit since then:
https://github.com/jmapio/jmap/commits/master
I am hoping to use along with https://github.com/virtualmin/virtualmin-gpl (also in Perl) which uses Dovecot: https://doxfer.webmin.com/Webmin/Dovecot_IMAP/POP3_Server
Thanks!
Marc
This had some setup. I have four messages in a FastMail account. All were unread. Via JMAP, I can see that they were all unread. All good.
In FastMail, I read two of them. FastMail does the right thing and I can confirm it worked via IMAP:
. SELECT INBOX
* 4 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD $IsTrusted $NotJunk $Forwarded $Junk $X-FM-Device-Push)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD $IsTrusted $NotJunk $Forwarded $Junk $X-FM-Device-Push \*)] Ok
* OK [UNSEEN 3] Ok
* OK [UIDVALIDITY 1386137841] Ok
* OK [UIDNEXT 3129] Ok
* OK [HIGHESTMODSEQ 17365878007025304772] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
. OK [READ-WRITE] Completed
. FETCH 1:* (FLAGS)
* 1 FETCH (FLAGS (\Seen $X-ME-Annot-2))
* 2 FETCH (FLAGS (\Seen $X-ME-Annot-2))
* 3 FETCH (FLAGS ($X-ME-Annot-2))
* 4 FETCH (FLAGS ($X-ME-Annot-2))
. OK Completed (0.000 sec)
The flag changes aren't reflected via the proxy, including in the client.
In the getMailboxes
response:
{
"id" : "9",
"mayAddItems" : true,
"mayCreateChild" : true,
"mayDelete" : false,
"mayReadItems" : true,
"mayRemoveItems" : true,
"mayRename" : false,
"mustBeOnlyMailbox" : true,
"name" : "Inbox",
"parentId" : null,
"role" : "inbox",
"sortOrder" : 1,
"totalMessages" : 4,
"totalThreads" : 4,
"unreadMessages" : 4,
"unreadThreads" : 4
},
The individual messages all have isUnread: true
as well (not showing here, its loooooong).
Further, changing the isUnread
flag (by opening a message) doesn't do the right thing:
[
[
"setMessages",
{
"create" : {},
"destroy" : [],
"state" : "328",
"update" : {
"m76fdbd6d5" : {
"isUnread" : false
}
}
},
"0"
],
[
"getMailboxes",
{
"ids" : [
"9"
],
"properties" : [
"totalMessages",
"unreadMessages",
"totalThreads",
"unreadThreads"
]
},
"1"
]
]
[
[
"messagesSet",
{
"accountId" : "9e2e4652-3191-11e5-847c-0026b9fac7aa",
"created" : {},
"destroyed" : [],
"newState" : null,
"notCreated" : {},
"notDestroyed" : {},
"notUpdated" : {},
"oldState" : null,
"updated" : [
"m76fdbd6d5"
]
},
"0"
],
[
"mailboxes",
{
"accountId" : "9e2e4652-3191-11e5-847c-0026b9fac7aa",
"list" : [
{
"id" : "9",
"totalMessages" : 4,
"totalThreads" : 4,
"unreadMessages" : 4,
"unreadThreads" : 4
}
],
"notFound" : null,
"state" : "328"
},
"1"
]
]
So it claims the update has occurred, but that's not reflected in the mailboxes
response, and there's no change on the backend IMAP server either.
My "A & B" folder isn't coming through correctly
When accessing messages with UTF-8 quoted-printable encoded headers, the values exposed in the JSON response by the JMAP proxy contain wrongly encoded unicode characters.
The following subject header in the original IMAP message
Subject: =?UTF-8?Q?Bern:=20Vari=C3=A9t=C3=A9,=20Juliette=20&=20Meeressause?=
results in a response
[
"messages",
{
"accountId": "xxxxx",
"list": [
{
"headers": {
"Subject": "Bern: Variété, Juliette & Meeressause",
...
},
...
"subject": "Bern: Variété, Juliette & Meeressause",
...
}
]
},
"#1"
]
It appears that the decoded header value is treated as iso-8859-1 encoded and is then converted to utf-8 resulting in a double encoding.
getThreadUpdates is returning message ids instead of thread ids.
This also means if you say 'fetchRecords => true', it claims the items are not found (since its looking up thread.id using message.id).
Folder names with non-ascii characters are listed by the IMAP server with UTF-7 encoding. The JMAP spec says for the mailbox name property: "This may be any UTF-8 string of at least 1 character in length".
So converting UTF-7 folder names into UTF-8 is required here.
Example IMAP folder listing:
C: A0005 LIST (SUBSCRIBED) "" "*"
S: * LIST (\Noinferiors \Subscribed) "/" INBOX
S: * LIST (\Subscribed) "/" Archive
S: * LIST (\Subscribed) "/" Drafts
S: * LIST (\Subscribed) "/" Junk
S: * LIST (\Subscribed) "/" Sent
S: * LIST (\Subscribed) "/" Spam
S: * LIST (\Subscribed) "/" Test-&A5EDwAOsA70DxAO3A8MDtw-
S: * LIST (\Subscribed) "/" Trash
S: * LIST (\Subscribed) "/" "Shared Folders/shared/Project Planning"
S: * LIST (\Subscribed) "/" "Shared Folders/shared/testing"
S: A0005 OK Completed
JMAP response for getMailboxes
:
[
[
"mailboxes",
{
"accountId": "ce69e3d4-2430-11e6-810b-0242ac120007",
"list": [
...
{
"id": "50",
"mayAddItems": true,
"mayCreateChild": true,
"mayDelete": true,
"mayReadItems": true,
"mayRemoveItems": true,
"mayRename": true,
"mustBeOnlyMailbox": true,
"name": "Test-&A5EDwAOsA70DxAO3A8MDtw-",
"parentId": null,
"role": null,
"sortOrder": 3,
"totalMessages": 0,
"totalThreads": 0,
"unreadMessages": 0,
"unreadThreads": 0
},
...
],
"notFound": null,
"state": "24"
},
"#1"
]
]
This is what we do in Cyrus, and it makes things heaps faster when getting threads.
It seems that the public proxy running at proxy.jmap.io returns these values in the Attachment object:
"isInline":"1"
"isInline":""
Neither of them is valid JSON Boolean value as per the JMAP and JSON specifications.
Even if it succeed, mailbox creation returns an error like
[["error",{"message":"Can't use string (\"20\") as a HASH ref while \"strict refs\" in use at /home/jmap/jmap-perl/JMAP/API.pm line 374.\n","type":"serverError"},"#0"]]
Spec:
If an id given cannot be found, the update or destroy MUST be rejected with a notFound set error.
I ask for all the ids of my mailboxes:
~/code/topicbox$ ./jc getMailboxes | jq '.[0][1].list | .[] | .id'
"1"
"2"
"3"
"4"
"5"
"6"
"7"
"8"
"10"
"11"
"12"
So, I have mailboxes with ids 1 .. 12, but not 9. I'll try updating the name on 13.
~/code/topicbox$ ./jc setMailboxes '{update:{13:{name:"YourFace"}}}'
[
[
"mailboxesSet",
{
"accountId" : "11049eb6-1dcc-11e6-82c1-f23c91556942",
"created" : {},
"destroyed" : [],
"newState" : null,
"notCreated" : {},
"notDestroyed" : {},
"notUpdated" : {},
"oldState" : null,
"updated" : [
"13"
]
},
"a"
]
]
Seems to work, but:
~/code/topicbox$ ./jc getMailboxes '{ids:["13"]}'
[
[
"mailboxes",
{
"accountId" : "11049eb6-1dcc-11e6-82c1-f23c91556942",
"list" : [],
"notFound" : [
"13"
],
"state" : "16"
},
"a"
]
]
When I try send email from Drafts to Outbox:
[ [ "setMessages", { "update": { "md5a07f187":{ "mailboxIds":["5"] } } }, "#1" ] ]
, I get error:
[ [ "error", { "message": "isa check for \"helo\" failed: undef is not a string! at (eval 310) line 108.\n\teval {...} called at (eval 310) line 107\n\tEmail::Sender::Transport::SMTPS::new(\"Email::Sender::Transport::SMTPS\", HASH(0x2bee918)) called at /home/jmap/jmap-perl/JMAP/Sync/Standard.pm line 103\n\tJMAP::Sync::Standard::send_email(JMAP::Sync::Standard=HASH(0x66c9b00), \"From: bob\\@bicsh.com\\x{d}\\x{a}To: alice\\@bicsh.com\\x{d}\\x{a}Cc: \\x{d}\\x{a}Bcc: \\x{d}\\x{a}Subjec\"...) called at /home/jmap/jmap-perl/JMAP/ImapDB.pm line 135\n\tJMAP::ImapDB::backend_cmd(JMAP::ImapDB=HASH(0x5969338), \"send_email\", \"From: bob\\@bicsh.com\\x{d}\\x{a}To: alice\\@bicsh.com\\x{d}\\x{a}Cc: \\x{d}\\x{a}Bcc: \\x{d}\\x{a}Subjec\"...) called at /home/jmap/jmap-perl/JMAP/ImapDB.pm line 1109\n\tJMAP::ImapDB::update_messages(JMAP::ImapDB=HASH(0x5969338), HASH(0x23b21c8), CODE(0x66f0160)) called at /home/jmap/jmap-perl/JMAP/API.pm line 1178\n\tJMAP::API::setMessages(JMAP::API=HASH(0x66c9a28), HASH(0x23b21f8), \"#1\") called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 601\n\teval {...} called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 601\n\tJMAP::Backend::handle_jmap(JMAP::ImapDB=HASH(0x5969338), ARRAY(0x23be420), \"#0\") called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 201\n\teval {...} called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 187\n\tJMAP::Backend::__ANON__(AnyEvent::Handle=HASH(0x6489048), ARRAY(0x23eb1f0)) called at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1735\n\tAnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x6489048)) called at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1310\n\tAnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x6489048)) called at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2009\n\tAnyEvent::Handle::__ANON__(EV::IO=SCALAR(0x23b2168), 1) called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 120\n\teval {...} called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 120\n\tJMAP::Backend::process_request(JMAP::Backend=HASH(0x1d76a60), Net::Server::Proto::TCP=GLOB(0x63d0060)) called at /usr/local/share/perl/5.20.2/Net/Server.pm line 74\n\tNet::Server::run_client_connection(JMAP::Backend=HASH(0x1d76a60)) called at /usr/local/share/perl/5.20.2/Net/Server/Fork.pm line 199\n\tNet::Server::Fork::run_client_connection(JMAP::Backend=HASH(0x1d76a60)) called at /usr/local/share/perl/5.20.2/Net/Server/Fork.pm line 144\n\tNet::Server::Fork::loop(JMAP::Backend=HASH(0x1d76a60)) called at /usr/local/share/perl/5.20.2/Net/Server.pm line 60\n\tNet::Server::run(\"JMAP::Backend\", \"host\", \"127.0.0.1\", \"port\", 5000) called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 124\n", "type": "serverError" }, "#1" ] ]
When I connect to this SMTP server with different client, it works. Can somebody help me, where can be problem.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.