foxcpp / maddy Goto Github PK
View Code? Open in Web Editor NEW✉️ Composable all-in-one mail server.
Home Page: https://maddy.email
License: GNU General Public License v3.0
✉️ Composable all-in-one mail server.
Home Page: https://maddy.email
License: GNU General Public License v3.0
There could be several filter backends. One of them could be https://github.com/emersion/go-milter
where can I find the test mail I sent?
how to set up an account?
This is necessary when upstream rejects messages.
Generally, we should try to not introduce a lot of C dependencies (#20). We can implement basic support for local authentication on Unix systems using /etc/passwd and /etc/shadow databases.
My initial idea is to inspect Received fields in the headers of the messages.
If it contains value mentioning the recipient address we are handling and our hostname - we bounce the message because it was processed by our server already.
It is important to check hostname because a single recipient may be processed by multiple servers and each will include its own Received field.
Another consideration to take into account: We should skip Received fields we can't process as some MTAs include information using non-standard formats (qmail, for example). Actually, SMTP RFC requires MTA to not fail delivery if an invalid Received field is present in the header.
Postfix relies on Delivered-To (non-standard AFAIK) fields for this purpose. We might want to include this field when the message leaves the pipeline. Currently, it is included only when the message is stored in an IMAP mailbox.
Steps to reproduce:
Before that, we need to get DKIM verification working.
I guess it might make sense to implement server-side encryption for mail storage to protect past messages in case of database compromise. PGP is a ready-to-use public key[1] infrastructure[2], though we might want to use keys generated by the server for each user to avoid all trust issues.
The symmetric key used to protect private key can be derived from user password using the computation-intensive hash algorithm such as bcrypt.
[1] Public key encryption is necessary to ensure that we can deliver new messages without knowledge of the decryption key.
[2] Tho I think we might want to pick something simpler for this purpose, ideas?
There are also some problems that need to be solved:
Originally posted by @foxcpp in #10 (comment)
By default configuration should be read from /etc/maddy/Maddyfile
.
All state (queues, databases) should be stored in $MADDYPATH
with the default value of /var/lib/maddy
.
The current implementation is just a placeholder using the current working directory for everything.
As an SMTP pipeline step:
proxy address0 address1 address2
Addresses use the same syntax as config definitions for endpoint modules. Remotes are tried in specified order, no more remotes will be tried after a successful transaction.
Hi,
I'd like to use this project for my ajaj ajaj
comment je jajjjaj
Currently it is basically line-oriented and thus will not handle the following cases as expected:
Closing brace will be ignored, leading to incorrect parsing of following statements.
name args { directive }
name args {
directive }
This will fail with parse error.
name args
{
directive
}
Just wanted to ask, is there a way to specify that Maddy would serve the right cert based on SNI when using TLS?
Hi,
the current code does not compile. Most certainly because of an updated underlying library.
If I understand what is going on, it seems the go-imap-proxy code tries to use (go-imap-proxy/mailbox.go:16) the m.u.c.Mailbox function as a struct. So go-imap-proxy seems to be the culprit here.
That is the first error. Did not look at the others though.
Are you still interested by that code ? (ie: would you be interested in some help ?)
When I try to compile from a fresh clone I get the following errors:
# github.com/emersion/go-imap-proxy
../../go-imap-proxy/mailbox.go:16: m.u.c.Mailbox.Name undefined (type func() *imap.MailboxStatus has no field or method Name)
../../go-imap-proxy/mailbox.go:33: m.u.c.Mailbox.Name undefined (type func() *imap.MailboxStatus has no field or method Name)
../../go-imap-proxy/mailbox.go:34: invalid indirect of m.u.c.Mailbox (type func() *imap.MailboxStatus)
../../go-imap-proxy/mailbox.go:38: cannot use items (type []string) as type []imap.StatusItem in argument to m.u.c.Status
../../go-imap-proxy/mailbox.go:68: cannot use items (type []string) as type []imap.FetchItem in argument to m.u.c.UidFetch
../../go-imap-proxy/mailbox.go:70: cannot use items (type []string) as type []imap.FetchItem in argument to m.u.c.Fetch
../../go-imap-proxy/mailbox.go:103: cannot use string(operation) (type string) as type imap.StoreItem in argument to m.u.c.UidStore
../../go-imap-proxy/mailbox.go:105: cannot use string(operation) (type string) as type imap.StoreItem in argument to m.u.c.Store
../../go-imap-proxy/user.go:34: cannot use mailbox literal (type *mailbox) as type backend.Mailbox in append:
*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
../../go-imap-proxy/user.go:54: impossible type assertion:
*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
../../go-imap-proxy/user.go:54: too many errors
# github.com/emersion/go-pgpmail/imap
../../go-pgpmail/imap/mailbox.go:26: cannot use item (type string) as type imap.FetchItem in argument to imap.ParseBodySectionName
../../go-pgpmail/imap/mailbox.go:32: cannot use items (type []string) as type []imap.FetchItem in argument to m.Mailbox.ListMessages
../../go-pgpmail/imap/mailbox.go:64: cannot use items (type []string) as type []imap.FetchItem in argument to m.Mailbox.ListMessages
../../go-pgpmail/imap/user.go:24: cannot use u.getMailbox(m) (type *mailbox) as type backend.Mailbox in assignment:
*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
../../go-pgpmail/imap/user.go:34: cannot use u.getMailbox(m) (type *mailbox) as type backend.Mailbox in return argument:
*mailbox does not implement backend.Mailbox (wrong type for ListMessages method)
have ListMessages(bool, *imap.SeqSet, []string, chan<- *imap.Message) error
want ListMessages(bool, *imap.SeqSet, []imap.FetchItem, chan<- *imap.Message) error
scdoc is probably the way to go.
Look-up MTA-STS policy for remote MTAs before connecting to them, enforce TLS and cache policy.
smtp://127.0.0.1:smtp {
local
destination example.org {
proxy lmtp:///tmp/upstream.socket
}
}
We want to run certain checks (#41) on the envelope before the message body is sent to us to not waste traffic. The current pipeline implementation doesn't allow to run steps before the message body is received.
Also, SMTP doesn't allow us to "abort" transaction once we sent 354 intermediate reply to the DATA command.
I see two possible solutions for this problem:
Defined somehow like that:
smtp ... {
envelope_checks {
verify_source_hostname
verify_source_mx
...
}
}
The checks are executed when RCPT/MAIL FROM command received and use the following interface:
type EnvelopeCheck interface {
CheckFrom(ctx *DeliveryContext, from string) error
CheckRcpt(ctx *DeliveryContext, rcpt string) error
}
Corresponding method is called for each recipient and for source.
Once the client issues the DATA command, start executing pipeline steps. If any of the steps need the message body - send 354 reply and pass body reader to the pipeline step (with some tricks it can be made fully transparent for step implementation).
destination
directive)Using a "550 mailbox not found" (or
equivalent) reply code after the data are accepted makes it difficult
or impossible for the client to determine which recipients failed.
Question regarding standards conformance/interoperability: Are we allowed to return arbitrary failures in the DATA command before the client sends us the body?
--
Original post:
Let's say I configured IMAP endpoint as follows (where first line creates IMAP backend):
imap://127.0.0.1:1993 {
sql sqlite3 maddy.db
}
Then I want to have SMTP endpoint that will deliver mail to same storage. How would I do this?
smtp://127.0.0.1:1025 {
sql sqlite3 maddy.db
}
This approach creates another set of problems, because it now requires two separate backend/upstream objects to coordinate access to the same storage (think of IMAP unilateral updates).
Global variables? External IPC sockets? All this seems to be dirty solution.
What is we can create one "storage" object and associate it with multiple IMAP/SMTP endpoints?
This will transform "another set of problems" into just serialization of access to storage object. Which is easily solved by throwing some mutexes into it (or even without them, I haven't tested that but go-sqlmail backend object should be safe for concurrent use by multiple goroutines).
It also reduces resources usage (we will have only one SQLite "connection" page cache, for example)
Now I can imagine something like this:
backend sql arbitrary_name {
driver sqlite3
dsn maddy.db
}
imap://127.0.0.1:1993 {
backend arbitrary_name
# of course this requires storage object to implement go-imap's Backend interface
}
smtp://127.0.0.1:1025 {
backend arbitrary_name
# and also go-smtp's Backend here now.
}
What do you think?
Global directive?
log <targets...>
log stderr
Write to stderr (default).
log file_name
Write to file.
log syslog
Send messages to local syslog daemon.
Can be combined:
log stderr syslog /var/log/maddy/maddy.log
mais on lanse comen XD
username+anything@domain should be rewritten to username@domain during local message delivery.
--
Once sieve filtering has been added, is the possibility of having a [email protected] email that gets emails forwarded into spam folder a reality?
Also +1 on having ManageSieve support too.Sorry about my English, it's my native language.
Originally posted by @NamedKitten in #53 (comment)
I remember having a use-case where I wanted to give email accounts to all PAM users but use passwords from a separate database. And also people on IRC channels for dovecot and postfix often ask questions regarding weird mixes of authentication data sources. So I guess we should have a generic tool to handle such cases.
We create the multi
module that implements authentication provider interface (so it can be used in SMTP, IMAP, etc):
multi instance_name {
user pam
user virtual { file /etc/maddy/userlist }
pass virtual { file /etc/maddy/passwd }
}
user
directive here refers to a module implementing the following interface:
type UserDB interface {
HasUser(name string) bool
}
pass
directive refers to an authentication provider.
If there is at least one user
directive - then at least one "userdb" module should say that the user exists.
Then the user password is also checked against providers listed using pass
directives, at least one provider should accept the password.
multi
module itself also implements the UserDB interface, this allows mixing things together in more complicated use-cases to get the right results.
Allow writing module definitions inside other module definitions, omitting names.
This feature would allow writing less verbose configuration files with fewer entities (so you don't have to look around a lot to understand what is happening). The key point is to require "named entities" only when they are shared between server components (modules).
For example, the following configuration:
hostname example.org
sql {
driver sqlite3
dsn /var/lib/maddy/all.db
}
smtp smtp://0.0.0.0:25 {
pipeline incoming_smtp
}
smtp smtp://0.0.0.0:587 {
pipeline outgoing_smtp
}
# let's pretend that we have two modules for DKIM signing and verification, dkim_sign and dkim_verify
dkim_sign {
public_key pubkey_file_path
private_key privkey_file_path
}
pipeline incoming_smtp {
filter dkim_verify
filter milter /var/run/spamassasin.sock
deliver sql
}
pipeline outgoing_smtp {
filter dkim_sign
deliver incoming_smtp local_only # let's pretend that pipeline implements DeliveryTarget
deliver queue remote_only
}
queue {
max_tries 1
target remote
}
could be written like that:
hostname example.org
sql {
driver sqlite3
dsn /var/lib/maddy/msgs.db
}
smtp smtp://0.0.0.0:25 {
pipeline {
filter dkim_verify require # refering to singleton module 'dkim_verify'
filter milter /var/run/spamassasin.sock # refering to singleton module 'milter'
deliver sql # refering to 'sql' module instance
}
}
smtp smtp://0.0.0.0:587 {
auth pam # refering to singleton module 'pam'
pipeline {
filter dkim_sign { # inline-defined instance of module dkim_sign
public_key pubkey_file_path
private_key privkey_file_path
}
deliver queue { # inline-defined instance of module queue
max_tries 1
target remote # refering to singleton module 'remote' here
}
deliver smtp
}
}
Generic syntax change is from
{
operation module_instance_name module_options
}
to
{
operation module_name {
module_config_directives
}
}
Here "inner" pipeline instance gets some artificial name like parentname_N
so it can use it as a key to store persistent data.
!!! Reordering of configuration directives here may change "artificial names". Can we do something about it?
!!! We need a different way to specify per-call options
Perhaps
{
operation module_name {
module_config_directives
opts options_go_here
}
}
Note: a { dir } b c
is parsed as
a {
dir
}
b c
But I'm not very sure about the whole idea. Any thougnts?
Configuring MUAs is made much easier with support for automatic configuration. Mozilla's de-facto standard for this is quite widely used: https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration
I'm currently manually maintaining one of these at https://autoconfig.ur.gs/.well-known/autoconfig/mail/config-v1.1.xml
There are DNS-based standards too, but I don't think they're as widely used.
We'll need a HTTP(S) listener for automatic TLS, activesync, webmail, etc, support too, so I don't think it's a big deal to add one.
Following conditions should be met for verification to pass:
These checks are built on the assumption that we have a DNSSEC-enabled resolver and the source server does have DNSSEC enabled.
After emersion/go-imap#232 gets resolved we want to update maddy to log IP addresses for failed login attempts.
We already do that for SMTP so no changes required here.
Also, write an article (perhaps on wiki pages since it is not related directly to maddy) about how to configure fail2ban for it. Before that, we need to do #26.
Perhaps something like that:
smtp ... {
aliases path_to_file_with_mappings
}
Format same as postfix's text representation (http://www.postfix.org/aliases.5.html). Except probably we will leave |command
, /file
, :include
support out for now.
It would be nice to support DANE in addition to MTA-STS to increase interoperability.
Again, there are two sides in DANE support:
This allows for multiple workers and makes sure no message gets lost.
Just like Caddy.
This will allow to reuse a lot of components developed for PAM.
Some example code can be found here: https://stackoverflow.com/questions/10910193/how-to-authenticate-username-password-using-pam-w-o-root-privileges
Except probably we want to define our own PAM service instead of using "su".
Pipeline and filters code copies messages multiple times, resulting in memory usage spikes when processing messages with attachment
One of the possible solutions: Add type-assertions where necessary and reuse existing copy instead of ioutil.ReadAll'ing it.
Type-assert on an interface with Bytes() method [1] when byte slice is needed (go-imap-sql with in-DB storage, e.g. SQLite)
Type-assert on io.ReadSeeker when the message body needs to be sent to multiple locations.
[1] bytes.Reader does not allow to obtain underlying byte slice, we might need to wrap it to add Bytes() method.
Some filters only read the message and prepend headers, but the current design forces them to rewrite and copy around the entire message body
One of the possible solutions: Special-case prepend operation and optimize it.
One of the possible solutions: Store header in parsed form and allow filters to manipulate it. Serialize when the message exits pipeline (e.g. gets delivered to storage or milter[2]).
[2] Tho it seems like the milter protocol allows us to avoid reserialization of headers.
We need to submit message content to multiple sources (including milters and other out-of-process filters). This basically forces us to buffer the entire message. Currently, we do this in memory. This leads to increased memory usage, especially when delivering messages with attachments.
One of the possible solutions: Allow to buffer messages on disk instead of RAM.
This variant might be incompatible with the second solution for the second problem.
Probably maildir
Just wanted to ask, is every component compatible with internationalized domain names? I'd really like to leave Postfix because it does database connections LATIN-1 only which is rather unusable.
We can have other means to submit messages for processing: IMAP submission extensions, JMAP, etc.
This will also simplify the code of the SMTP endpoint module a lot.
One module - one job. The job of the SMTP endpoint module is to accept messages using SMTP and push them to the pipeline. The job of the pipeline module is to connect modules to each other.
value operator value
Value can be either variable prefixed with $
or
Variable can be all fields from DeliveryContext and also any key from DeliveryContext.Ctx. If the variable doesn't exist - its value assumed to be an empty string.
The operator is one of the following:
=
Variable value must be exactly equal to the value in condition.=*
Similar to the =
but is case-insensitive.!=
Inverted =
.!=*
Inverted =*
.~
Value is a regexp that should match variable value.~*
Case-insensitive version of ~
.!~
Inverted ~
.!~*
Inverted ~*
Inspired by the nginx's if. I highlighted operations currently implemented for the match
block.
Variable value is converted into string before comparsion.
$rcpt_domain = "emersion.fr"
$src_ip ~ "^192\.168\..+"
match
=> if
Rename match
block into if
and make it use conditions syntax described above.
if $rcpt_domain = "emersion.fr" {
proxy smtp://0.0.0.0
}
E.g. [email protected] and [email protected] are the same.
Messages not getting delivered to the local storage should not have recipient addresses case-folded tho.
Main Use-Case: Use different databases for different domains.
Proposed Solution: Introduce module perdomain
that calls different auth. providers depending on the source domain and implement auth. provider interface itself.
auth perdomain {
domain example.org {
auth virtual { file /etc/maddy/example.org-users }
}
domain example.com {
dont_strip_domain # by default perdomain removes domain from username before passing it to the "nested" provider
auth virtual { file /etc/maddy/example.com-users }
}
# Both make authentication fail by default.
unknown { # domain not matched by any directives.
...
}
nodomain { # no domain present in username
...
}
}
Rationale and possible alternative: #58 (comment)
Make it possible to specify multiple cert-key pairs in the tls
directive:
tls <cert> <key> <cert> <key>
Load them all then call tls.Config.BuildNameToCertificate
.
The relevant code is in internal/config/tls_server.go in the readTLSBlock function.
Documentation to update: docs/man/maddy-imap.5.scd, docs/man/maddy-smtp.5.scd and probably docs/man/maddy-tls.5.scd.
Perhaps something like that:
tls perdomain {
<domain> <cert_file> <key_file>
_default cert_file key_file
}
Possible using GetCertificate or GetConfigForClient callbacks in tls.Config.
Things like DKIM, MTA-STS are tightly coupled with DNS and HTTPS.
Should we become tiny DNS/HTTPS servers too?
RCPT TO:<postmaster>
Should be accepted as valid.
There are several places around the code that assume user@domain in recipients addresses, need to check them and probably move some code around.
See https://tools.ietf.org/html/rfc5321#page-14 and https://tools.ietf.org/html/rfc5321#section-4.5.1.
Messages are delivered multiple times to local mailboxes.
Investigation revealed that sometimes maddy fails to acknowledge message delivery and instead just doesn't reponds with anything despite successfull delivery. Remote MTA timeouts and retries delivery later resulting in duplicates.
Server restart fixes this problem, but only for some time. It happens only for some messages (needs to be additionally checked) and always reproduced when corresponding SMTP session is replayed as-is.
maddy with experimental queue implementation (824fbdc). Based on this commit from this repository: 2355cd9
Using go-imap-sql with PostgreSQL driver.
CC @NamedKitten
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.