Gevent SMTP Server with MongoDB storage
Demo:
- URL: http://188.165.254.60:8083
- Login: [email protected]
- Password: password
Features:
- SMTP Server high performance with Gevent Coroutine ( fork of Gsmtpd )
- Postfix XFORWARD extension
- Record messages in MongoDB
- Ability to use a custom python plugin to edit the message before recording
The use in production is not yet guaranteed
Table of Contents
Mode for quarantine or statistics only
- Network: smtp sender or recipient (with internet or local network)
- Content Filter: amavisd-new with xxx_quarantine_to (smtp:) parameters
- Configuration in amavisd.conf
- Zero configuration for Postfix
Mode for statistics or honey pot
- Network: smtp sender or recipient (with internet or local network)
- Optional: filtering after delivery to postfix by mongo mail
Mode for spam/virus filtering and statistics
- Network: smtp sender or recipient (with internet or local network)
- Filtering: clamav/spamassassin with TCP connection (without amavisd-new)
- Out filter: delivered so clean else
- Message is not transformed (unless in quarantine mode if MMS_REAL_RCPT=1)
- Gross message is stored in GridFS after being compressed (zlib) and converted to base64
- In proxy mode, the raw message is sent before the registration in MongoDB and is not saved if an error occurs while sending or all recipients are rejected
- Docker 1.4.1
- Ubuntu 14.04
- MongoDB 2.6.5
- Python 2.7.6
- Gevent 1.0
- Pymongo2_8 2.8 and Pymongo 3.0
- Postfix 2.5.5
- Amavisd-new 2.6.4
{'client_address': '139.129.236.68',
'message': ObjectId('55252ae62d4b25262070a176'),
'rcpt': ['[email protected]'],
'rcpt_count': 1,
'rcpt_refused': {},
'received': datetime.datetime(2015, 4, 8, 13, 19, 34, 579000, tzinfo=tzutc()),
'sender': '[email protected]',
'server': '127.0.0.1',
'store_key': '77bd8b356cf2c593e61a6c0a7cbc5572eb357a7b857adca402ee40021db34fa6',
'xforward': {'ADDR': '139.129.236.68',
'HELO': 'mx.example.org',
'NAME': 'mx.example.org'}}
'message': ObjectId('55252ae62d4b25262070a176') is reference to data in Gridfs
{'_id': ObjectId('55252ae62d4b25262070a178'),
'client_address': u'139.129.236.68',
'completed': 0,
'errors_count': 0,
'events': [],
'files': [],
'files_count': 0,
'group_name': u'DEFAULT',
'headers': {},
'internal_field': 0,
'is_banned': 0,
'is_bounce': 0,
'is_in': 1,
'is_spam': 0,
'is_unchecked': 0,
'is_virus': 0,
'mark_for_delete': 0,
'message': ObjectId('55252ae62d4b25262070a176'),
'parsing_errors': [],
'queue': 1,
'rcpt': [u'[email protected]'],
'rcpt_count': 1,
'rcpt_refused': {},
'received': datetime.datetime(2015, 4, 8, 13, 19, 34, 579000, tzinfo=<bson.tz_util.FixedOffset object at 0x02B54E10>),
'sender': u'[email protected]',
'server': u'127.0.0.1',
'size': 0L,
'store_key': u'77bd8b356cf2c593e61a6c0a7cbc5572eb357a7b857adca402ee40021db34fa6',
'tags': [],
'xforward': {u'ADDR': u'139.129.236.68',
u'HELO': u'mx.example.org',
u'NAME': u'mx.example.org'}}
{'_id': ObjectId('55252ae62d4b25262070a178'),
'client_address': u'139.129.236.68',
'completed': 1,
'country': u'CN',
'errors_count': 0,
'events': [],
'files': [],
'files_count': 0,
'group_name': u'DEFAULT',
'headers': {u'Content-Transfer-Encoding': [u'base64', {}],
u'Content-Type': [u'text/plain', {u'charset': u'utf-8'}],
u'Date': u'Wed, 08 Apr 2015 13:19:34 UTC',
u'From': u'"Bertrand Auger" <[email protected]>',
u'Message-Id': u'<20150408131934.10264.63423@admin-VAIO>',
u'Mime-Version': u'1.0',
u'Subject': u'Provident tempora ad quasi enim in ratione excepturi. Optio soluta culpa voluptas labore in. Voluptatem aliquid est rerum in est adipisci dolore.',
u'To': u'"Thierry Leleu" <[email protected]>',
u'X-Mailer': u'MessageFaker'},
'internal_field': 0,
'is_banned': 0,
'is_bounce': 0,
'is_in': 1,
'is_spam': 0,
'is_unchecked': 0,
'is_virus': 0,
'mark_for_delete': 0,
'message': ObjectId('55252ae62d4b25262070a176'),
'message_id': u'20150408131934.10264.63423@admin-VAIO',
'parsing_errors': [],
'queue': 1,
'rcpt': [u'[email protected]'],
'rcpt_count': 1,
'rcpt_refused': {},
'received': datetime.datetime(2015, 4, 8, 13, 19, 34, 579000, tzinfo=<bson.tz_util.FixedOffset object at 0x02AC4E10>),
'sender': u'[email protected]',
'sent': datetime.datetime(2015, 4, 8, 13, 19, 34, tzinfo=<bson.tz_util.FixedOffset object at 0x02AC4E10>),
'server': u'127.0.0.1',
'size': 636L,
'store_key': u'77bd8b356cf2c593e61a6c0a7cbc5572eb357a7b857adca402ee40021db34fa6',
'subject': u'Provident tempora ad quasi enim in ratione excepturi. Optio soluta culpa voluptas labore in. Voluptatem aliquid est rerum in est adipisci dolore.',
'tags': [],
'xforward': {u'ADDR': u'139.129.236.68',
u'HELO': u'mx.example.org',
u'NAME': u'mx.example.org'}}
Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 X-Mailer: MessageFaker Message-ID: <20150408131934.10264.63423@admin-VAIO> From: "Bertrand Auger" <[email protected]> To: "Thierry Leleu" <[email protected]> Subject: Provident tempora ad quasi enim in ratione excepturi. Optio soluta culpa voluptas labore in. Voluptatem aliquid est rerum in est adipisci dolore. Date: Wed, 08 Apr 2015 13:19:34 UTC U2l0IHZvbHVwdGF0ZSByZXJ1bSBjb3Jwb3JpcyBkb2xvcmlidXMgZW9zLiBRdWFzIGVvcyBub24g bW9kaSBxdWlzLiBBbGlhcyB2ZWwgbGF1ZGFudGl1bSBtYWduaSBzdXNjaXBpdC4gRnVnaWF0IGV0 IHF1aXMgZXQgaW4gYWNjdXNhbXVzLg==
- MongoDB Server
- Postfix or Amavisd-new
- Python 2.7.6+ (< 3.x)
- python-gevent 1.0+
- recent setuptools and pip installer
$ pip install mongo-mail-server
$ mongo-mail-server --help
- Docker 1.4+
- MongoDB Server
Contenair based on Ubuntu 14.04 - Python 2.7
Image from Dockerfile
$ docker pull dockerfile/mongodb
$ docker run -d -p 27017:27017 --name mongodb dockerfile/mongodb mongod --smallfiles
# Persist mongodb
$ docker run -v /home/persist/mongodb:/data/db -d -p 27017:27017 --name mongodb dockerfile/mongodb mongod --smallfiles
$ git clone https://github.com/radical-software/mongo-mail-server.git
$ cd mongo-mail-server && docker build -t mongo-mail-server .
# help and verify
$ docker run -it --rm mongo-mail-server --help
$ mongodb_ip=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' mongodb)
# start for test
$ docker run -it --rm -e MMS_MONGODB_URI=mongodb://$mongodb_ip/message -p 172.17.42.1:14001:14001 mongo-mail-server
# start of background (optional: bind of docker0 interface)
# Add --restart=always for automatic restart
$ docker run -d --name mms -e MMS_MONGODB_URI=mongodb://$mongodb_ip/message -p 172.17.42.1:14001:14001 mongo-mail-server
# Logs
$ docker logs mms
2015-02-12 07:35:36 rs_smtpd_server: [INFO] - Starting SMTP Server - server[mongo-quarantine] - on 0.0.0.0:14001 (PID:1)
Server mode: mongo-quarantine | mongo-proxy | mongo-proxy | debug
Default: mongo-quarantine
# with command mode
$ export MMS_SERVER=mongo-quarantine
# with docker environ
$ docker run -e MMS_SERVER=mongo-quarantine
# with command arguments
$ mongo-mail-server --server mongo-quarantine
Host bind
Default: 0.0.0.0
# with command mode
$ export MMS_HOST=0.0.0.0
# with docker environ
$ docker run -e MMS_HOST=0.0.0.0
# with command arguments
$ mongo-mail-server --host 0.0.0.0
Port bind
Default: 14001
# with command mode
$ export MMS_PORT=14001
# with docker environ
$ docker run -e MMS_PORT=14001
# with command arguments
$ mongo-mail-server --port 14001
Default: mongodb://localhost/message
http://docs.mongodb.org/manual/reference/connection-string/
# with command mode
$ export MMS_MONGODB_URI=mongodb://localhost/message
# with docker environ
$ docker run -e MMS_MONGODB_URI=mongodb://localhost/message
# with command arguments
$ mongo-mail-server --mongo-host mongodb://localhost/message
DB Name for recording mails
Default: message
# with command mode
$ export MMS_MONGODB_DATABASE=message
# with docker environ
$ docker run -e MMS_MONGODB_DATABASE=message
# with command arguments
$ mongo-mail-server --mongo-database message
Collection Name for recording mails
Default: message
# with command mode
$ export MMS_MONGODB_COLLECTION=message
# with docker environ
$ docker run -e MMS_MONGODB_COLLECTION=message
# with command arguments
$ mongo-mail-server --mongo-collection message
Timeout for smtp transaction from Postfix
Default: 600 (seconds)
Size limit of message (in bytes)
Default: 0 (no limit)
Replace smtp recipient by real recipients (for quarantine with amavisd-new)
Default: disable
# with command mode
$ export MMS_REAL_RCPT=1
# with docker environ
$ docker run -e MMS_REAL_RCPT=1
# with command arguments
$ mongo-mail-server --real-rcpt
caution
Before amavisd-new 2.7.0 the recipient envelope is replaced by xxx_quarantine_to parameters Starting from 2.7.0, use macro '%a' in xxx_quarantine_to parameters
caution
About IP Address of smtp sender: Amavis does not use the extension SMTPD FORWARD to send mails in quarantine. The original IP address is lost. The solution might be to use postfix to amavis output for quarantine and postfix then return the message to mongo-mail
$ vi amavisd.conf
# ip address and port of Mongo Mail Server
$archive_quarantine_method = 'smtp:[172.17.42.1]:14001';
# Any valid email address. Domain few not exist
$archive_quarantine_to = '[email protected]';
# reload amavis
$ vi amavisd.conf
$archive_quarantine_method = 'smtp:[172.17.42.1]:14001';
$archive_quarantine_to = '[email protected]';
$virus_quarantine_method = $archive_quarantine_method;
$banned_files_quarantine_method = $archive_quarantine_method;
$spam_quarantine_method = $archive_quarantine_method;
# Not quarantine for clean mail - already stored with archive_quarantine_method
$clean_quarantine_method = undef;
# Not quarantine for bad header mail
$bad_header_quarantine_method = undef;
$virus_quarantine_to = $archive_quarantine_to;
$banned_quarantine_to = $archive_quarantine_to;
$spam_quarantine_to = $archive_quarantine_to;
#OR
$virus_quarantine_to = '[email protected]';
$banned_quarantine_to = '[email protected]';
$spam_quarantine_to = '[email protected]';
Dedicate a postfix server for this purpose
# main.cf - ip:port of Mongo Mail
smtpd_proxy_filter=127.0.0.1:14001
# or with command line
$ postconf -e 'smtpd_proxy_filter=127.0.0.1:14001'
# reload postfix
$ postix reload
The module must be in a package
# just required apply(metadata=None, data=None) method
# examples/plugins/dummy_plugin.py - modify server field and print message
import pprint
def apply(metadata=None, data=None):
metadata['server'] = "1.1.1.1"
pprint.pprint(metadata)
# Use:
$ mongo-mail-server --server debug --host 127.0.0.1 --port 14001 --plugin contrib.dummy_plugin start
# Use multiple plugins - run in the order of arguments
$ mongo-mail-server --server --plugin myplugin1 --plugin myplugin2 ...
# Use 172.17.42.1 is binding of docker0 else:
$ mms_ip=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' mms)
$ telnet $mms_ip 14001
Trying 172.17.1.19...
Connected to 172.17.1.19.
Escape character is '^]'.
220 a88632d9a311 SMTPD at your service
ehlo me.com
250-a88632d9a311 on plain
250-XFORWARD NAME ADDR PROTO HELO SOURCE PORT
250 HELP
XFORWARD NAME=mail.test.fr ADDR=1.1.1.1 HELO=test.fr
250 Ok
MAIL FROM:<[email protected]>
250 Ok
RCPT TO:<[email protected]>
250 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
Subject: Test
From: [email protected]
To: [email protected]
mytest
.
250 Ok: queued as ab80249748e0496b812b13c489a88002fbe102fc9c263b02a8b52101491f0128
QUIT
221 Bye
Connection closed by foreign host.
$ mongofiles -d message list
72c0f4898db56d5e10037e3f7f0c2af68704c8b86a2405d98a3e44e89bb56481 2188
571329a72c31a914251fd6fdecb160403345ee143c194cfc442ab5bee6118918 2188
a8de0206f9978346326cbcc9ffd5df647728268c19e8564dd1c2790b6c1404f3 2192
...
# Extract and write message to disk
$ mongofiles -d message get 75e3896c1c5d98a21fc14e9408e1b9be91ced60f2bc224416de63c975c9c2915
# Convert with python
python -c "import zlib,base64; print(str(zlib.decompress(base64.b64decode(open('75e3896c1c5d98a21fc14e9408e1b9be91ced60f2bc224416de63c975c9c2915', 'rb').read()))))"
# Parse to email.Message and print as_string()
python -c "import zlib,base64,email; print(email.message_from_string(str(zlib.decompress(base64.b64decode(open('75e3896c1c5d98a21fc14e9408e1b9be91ced60f2bc224416de63c975c9c2915', 'rb').read())))).as_string())"
Use MMS_TIMEOUT in environment or --timeout
Use MMS_DATA_SIZE_LIMIT in environment or --data-size-limit
>>> import os, zlib, base64
>>> from pprint import pprint as pp
>>> from email.parser import Parser, HeaderParser
>>> from pymongo import MongoClient
>>> from gridfs import GridFS
>>> client = MongoClient(os.environ.get('MMS_MONGODB_URI'))
>>> db = client['message']
>>> col = db['message']
>>> doc = col.find_one()
>>> fs = GridFS(db)
>>> msg_base64 = fs.get(doc['message']).read()
>>> msg_string = zlib.decompress(base64.b64decode(msg_base64))
>>> msg = Parser().parsestr(msg_string)
>>> msg
<email.message.Message instance at 0x7ff5e4054560>
- More tests
- Travis tests
- Monitoring with psutil
- Filter tasks
- Documentation of mongo-mail-reader command
- Documentation en Français
- Record to ElasticSearch
- Sends statistics to graphite, statsd, influxdb
To contribute to the project, fork it on GitHub and send a pull request, all contributions and suggestions are welcome.