heroku-log4mongo is a demo application showing how to obtain 240MB of free cloud logging in a mongodb for your Heroku deployments -- a sort of (very rough) free version of the Loggly addon, but with indefinite data retention.

To use mongodb for logging is not a new idea; the (I think original and nice) contribution of this sample code is to emphasize:

• the very nice log4mongo-python library making all this pretty easy,
• the awsome mongolab service, offering for free a 240MB database in the cloud (with a nice web interface);
• a nice hack to make gunicorn send its access log to the cloud,
• lasts but not least, how Heroku makes all this quite a piece of cake!

This demo applicaton is based on Flask but every other framework will work (albeit being less pleasurable to program).

The following part of this README describes how to use the demo application (locally and on Heroku), for more details on how the code works and the idea behind it, please check the blog post on my ProcrastinHacking blog.

To test this application, setup your Heroku environment as usual (assuming that the heroku and foreman gems are installed):

$ git clone git://github.com/mapio/heroku-log4mongo.git
$ heroku create --stack cedar

then add to it the free mongolab:starter addon to your app and set the VERSION:

$ heroku addons:add mongolab:starter
$ heroku config:add VERSION=production

that will give you a 240MB mongodb hosted in the cloud for free, as easy as pie! The VERSION variable is used to distinguish between the local and deployed versions of the application.

You can now setup a loccal testing environment using virtualenvwrapper:

$ mktmpenv; cd -
$ pip install -r requirements.txt

to use the app locally you need to setup the local .env file used by foreman to configure your app, in particular you need to store in it the URI of your databse, and the VERSION of the application with:

$ heroku config -s | grep 'MONGOLAB_URI' > .env
$ echo 'VERSION=developement' >> .env

(or conveniently use the ./scripts/set_env script). You are now ready to test locally: run the app and access its home with curl:

$ foreman start -p 8000
22:13:02 web.1     | started with pid 22744
22:13:02 web.1     | 2011-12-06 22:13:02 [22745] [INFO] Starting gunicorn 0.13.4
22:13:02 web.1     | 2011-12-06 22:13:02 [22745] [INFO] Listening at: http://0.0.0.0:8000 (22745)
22:13:02 web.1     | 2011-12-06 22:13:02 [22745] [INFO] Using worker: sync
22:13:02 web.1     | 2011-12-06 22:13:02 [22746] [INFO] Booting worker with pid: 22746
$ curl http://0.0.0.0:8000
Hello, world!
22:09:53 web.1     | 2011.12:06 22:09:53 [22691] [DEBUG/APPLICATION] This is an application log message
22:09:53 web.1     | 2011.12:06 22:09:53 [22691] [INFO/ACCESS] 127.0.0.1 - - [06/Dec/2011:22:09:53] "GET / HTTP/1.1" 200 13 "-" "curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5"

the first output lines being the normal app startup, followed by curl output, then a log message coming from the application, and finally a typical access log message.

You are now ready to try the application in production:

$ git push heroku master
$ heroku scale web=1
$ curl http://<YOUR_APP>.herokuapp.com/

where <YOUR_APP> is the app name you got as output of the heroku create command above. You can now get your logs from the cloud, for instance using the simple provided script:

$ ./scripts/get_logs
4.231.57.32 - - [06/Dec/2011:21:19:08] "GET / HTTP/1.1" 200 13 "-" "curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5"
$ ./scripts/get_logs -e
2011-12-06 21:15:19+00:00 Starting gunicorn 0.13.4
2011-12-06 21:15:19+00:00 Listening at: http://0.0.0.0:32652 (3)
2011-12-06 21:15:19+00:00 Using worker: sync
2011-12-06 21:15:19+00:00 Booting worker with pid: 10
$  ./scripts/get_logs -a
2011-12-06 21:19:08+00:00 This is an application log message

which will respectively output yuor access log, error log and application log, or you can access the nice web interface provided by the addon with:

$ heroku addons:open mongolab:starter

When you are down playing with this, don't forget to scale down:

$ heroku scale web=0

not to incur in unanticipated costs!

If you plan to use this in a real production application consider capped collections (a very high performance auto-FIFO database) as a simple way of controlling the size of your logs.