gliderlabs / logspout Goto Github PK

View Code? Open in Web Editor NEW

4.6K 4.6K 680.0 5.66 MB

Log routing for Docker container logs

License: MIT License

Makefile 5.39% Go 93.09% Shell 1.06% Dockerfile 0.47%

docker docker-logs syslog

logspout's People

Contributors

Stargazers

Watchers

Forkers

robxu9 ehazlett pinterb srid ngpestelos koanhealth defunctzombie live4code sttts ramtej carlanton sstarcher darkseed jongillies mdsol mcheriyath henrypfhu boucher stackengine remmelt jimmidyson synhaptein csquared baseboxorg rob-johnson bridg agileinsider sumologic charlesakalugwu nacyot happy-co derekelkins npateriya dansowter huskyeder haorui planitarinc wndhydrnt lclarkmichalek cobjet-dev jmcarbo nicocesar ruiaylin noyiba danielrowles-wf harlow gavbaa jsegura delectable raychaser gabrielnau getachew apostol7 leochencipher kanicz deet pixelated nagliyvred pavelvanecek shazow josharian brendangibat clever mkboudreau hbouvier altoplano kubatyszko remind101 ahjdzx t0mk b20n llamashoes caputomarcos cloudnautique tdfischer galal-hussein fpgeek mikemichel micahhausler tom-pryor carevoyance pugna0 lguminski iweave yqguodataman heidsoft-paas lytics dkapanidis bitxel glogiotatidis paddie josefsalyer jefflaplante macbash josh-padnick dkhunt27 dataman-cloud tomzhang sqdk chenrui2014

logspout's Issues

No http output at /logs in master

Using gliderlabs/logspout:latest, when I run docker run ... gliderlabs/logspout:latest and also docker run learn/ping ping github.com I get entries like this in :8000/logs

[1;37m        logspout|[martini] Started GET /logs for 192.168.59.3:53968�[0m
[1;36m    adoring_bell|PING github.com (192.30.252.128) 56(84) bytes of data.�[0m
[1;36m    adoring_bell|64 bytes from github.com (192.30.252.128): icmp_req=1 ttl=61 time=98.1 ms�[0m
[1;36m    adoring_bell|64 bytes from github.com (192.30.252.128): icmp_req=2 ttl=61 time=102 ms�[0m
[1;36m    adoring_bell|64 bytes from github.com (192.30.252.128): icmp_req=3 ttl=61 time=96.6 ms�[0m

�
However, using master and make dev I get no output from :8080/logs, although it's clear logspout is seeing the docker container lifecycle events (via console)

2015/03/24 13:46:27 loading and persisting routes in /mnt/routes
2015/03/24 13:46:27 logspout dev serving http on :8000
2015/03/24 13:47:03 event: 1203f24832b1 create
2015/03/24 13:47:03 event: 1203f24832b1 start
2015/03/24 13:47:10 event: 1203f24832b1 die

Startup routes, and filter documentation

Two parts:

How does one specify startup routes?
How do I do a "not this patter" in the route/filter system?

Use case:
I'm using this to forward logs to logstash where logstash is running in a container on the same server, so I don't want logstash parsing it's own logs.

Using --name for the Logspout container causes error

When I attempt to start logspout and specify a container name with --name a UDP error occurs

boyd@app-4:~ 16:20:31 >docker run -v=/var/run/docker.sock:/tmp/docker.sock progrium/logspout --name=logspout syslog://logs.papertrailapp.com:5555
2014/11/11 22:22:55 routing all to --name=logspout
2014/11/11 22:22:55 loading and persisting routes in /mnt/routes
2014/11/11 22:22:55 syslog: dial udp: missing address
boyd@app-4:~ 16:24:31 >

Support stdin for log source

This feature would allow commands running on the host to push logs to logspout by piping them to docker run logspout.

An example use case would be to run journalctl -f -o json | docker run -ti logspout

Logs appearing in the wrong order in papertrailapp.

If you use UDP then papertrailapp quite often shows the logs in the incorrect order. Would TCP be possible?

syslog: dial udp x.x.x.x:5000: resource temporarily unavailable

My container keeps crashing with the error:
syslog: dial udp x.x.x.x:5000: resource temporarily unavailable

But I am able to telnet to that IP using port 5000 with no problem. Is there a way to turn on more verbose logging to see what is happening?

Thank you.

stdout of container w/ TTY not captured

I have a centos:centos6 container running with /bin/bash -l and I'm trying to see echo "hi" logged to logstash through logspout.

I start the centos container like so: docker run --rm -it centos:centos6 /bin/bash -l and then I echo "hi"

I have ELK + Logspout running with the following docker-compose: https://gist.github.com/tobowers/b9eb3d7291399440e976#file-docker-compose-yml

We've tested sending syslog events to logstash and seeing them logged.

After logspout starts up, I would expect to see the stdout from the echo hi appearing as a syslog message in logstash, but I see nothing.

docker logs #{container_name} (where #{container_name} is the name of the centos6 container, correctly show the stdout from the bash commands.

Did I misinterpret the point of logspout? What am I doing wrong?

Attachment events, but no logs

I'm not sure if I'm running into a bug or if I'm doing something wrong. I attempted with the latest version of logspout in docker.io and I also rebuilt logspout from source.

docker run -v=/var/run/docker.sock:/tmp/docker.sock -itd -p 8000:8000 -e DEBUG=true --name logs progrium/logspout syslog://10.0.10.196:5140

Client version: 1.1.0
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): 79812e3
Server version: 1.1.0
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): 79812000

2014/09/29 10:53:32 attach: 5c08c7d0682d success
2014/09/29 10:53:32 attach: 431fdccfbe29 success
[debug] stdcopy.go:109 Error selecting output fd: (50)
2014/09/29 10:53:32 attach: 5c08c7d0682d finished
2014/09/29 10:53:32 attach: 244c3e51188e success
2014/09/29 10:53:32 routing all to syslog://10.0.10.196:5140
2014/09/29 10:53:32 loading and persisting routes in /mnt/routes
2014/09/29 10:53:32 logspout serving http on :8000
[debug] stdcopy.go:109 Error selecting output fd: (83)
2014/09/29 10:57:08 attach: 431fdccfbe29 finished
[martini] Started POST /routes for 172.17.42.1:46865
[martini] Completed 201 Created in 238.162us
[martini] Started GET /routes for 172.17.42.1:46870
[martini] Completed 200 OK in 277.06us
[martini] Started GET /logs for 172.17.42.1:46874
[martini] Completed 0  in 36.331076583s
[martini] Started GET /routes for 10.0.10.189:56731
[martini] Completed 200 OK in 107.044us
[martini] Started GET /logs for 10.0.10.189:56731

Error selecting output fd:

Hi !

I am trying to use logspout which sound exactly what we need on our docker project ...

However I currently am not able to see any log at all ...
I tried recompiling from source, etc ...

So, here is the error I have :

$ docker run --rm -e "DEBUG=true" -p 8000:8000 -v /tmp/route:/mnt/routes -v /var/run/docker.sock:/tmp/docker.sock logspout syslog://127.0.0.1:515
2014/07/15 15:35:39 routing all to syslog://127.0.0.1:515
2014/07/15 15:35:39 loading and persisting routes in /mnt/routes
2014/07/15 15:35:39 logspout serving http on :8000
[martini] Started GET /logs for 172.17.42.1:35016
2014/07/15 15:36:08 event: 286bb6e4361f create
2014/07/15 15:36:08 event: 286bb6e4361f create
2014/07/15 15:36:09 event: 286bb6e4361f start
2014/07/15 15:36:09 event: 286bb6e4361f start
2014/07/15 15:36:09 attach: 286bb6e4361f success
2014/07/15 15:36:09 attach: 286bb6e4361f success
[debug] stdcopy.go:112 Error selecting output fd: (50)
2014/07/15 15:36:12 attach: 286bb6e4361f finished
[debug] stdcopy.go:112 Error selecting output fd: (50)
2014/07/15 15:36:12 attach: 286bb6e4361f finished

Above, what you see is the launch of logspout, which goes smoothly, then launching another container (286bb6e4361f) ... and .... nothing else ...

Have you seen this ?

include syslog server to listen on udp/socket

Would be great to include a small go based syslog server library ( https://github.com/ziutek/syslog ) and allow via options to have logspout listen on udp syslog or unix socket.

this would allow pointing syslog in a container to logspout via udp or a volume mounted syslog socket. This would help apps that only support syslog logging run better in a container.

Loggly integration

Hello,

More of a question than an issue, is there a way to send data to match up with logglys format - https://www.loggly.com/docs/streaming-syslog-without-using-files/ ?

thanks

messages missing or duplicated

I'm running Docker and syslog-ng on a host machine (Ubuntu 12.04). I've configured syslog-ng on the host to receive messages over UDP on port 514 in the default source. The thought being I could have logspout send logs to the host machine's syslog-ng UDP port and then off to Papertrail over TLS.

So I ran logspout:

ubuntu@ip-10-80-72-198:~$ sudo docker run --name logspout --rm -v /var/run/docker.sock:/tmp/docker.sock -p 8000:8000 -h ip-10-80-72-198 progrium/logspout syslog://10.80.72.198:514
2014/09/12 20:56:52 routing all to syslog://10.80.72.198:514
2014/09/12 20:56:52 loading and persisting routes in /mnt/routes
2014/09/12 20:56:52 logspout serving http on :8000

Followed by a basic container:

ubuntu@ip-10-80-72-198:~$ sudo docker run --name redis.0 -d -p 10000:6379 dockerfile/redis
537e27cd7ec3aaad4bac716053d7d6cfe8a3438461f7b0a4b42d14aa4ee8192e

When running the container detached logs from Redis appear but are duplicated:

ubuntu@ip-10-80-72-198:~$ sudo tail -f /var/log/syslog
...
Sep 12 21:17:29 172.17.0.40 registrator[1]: 2014/09/12 21:17:29 registrator: added: 537e27cd7ec3 080a3c722ca2:redis.0:6379
Sep 12 21:17:29 172.17.0.40 redis.0[1]:                 _._
Sep 12 21:17:29 172.17.0.40 redis.0[1]:            _.-``__ ''-._
Sep 12 21:17:29 172.17.0.40 redis.0[1]:       _.-``    `.  `_.  ''-._           Redis 2.8.14 (00000000/0) 64 bit
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   .-`` .-```.  ```\/    _.,_ ''-._
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  (    '      ,       .-`  | `,    )     Running in stand alone mode
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |    `-._   `._    /     _.-'    |     PID: 1
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   `-._    `-._  `-./  _.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |`-._`-._    `-.__.-'    _.-'_.-'|
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |    `-._`-._        _.-'_.-'    |           http://redis.io
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   `-._    `-._`-.__.-'_.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |`-._`-._    `-.__.-'    _.-'_.-'|
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |    `-._`-._        _.-'_.-'    |
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   `-._    `-._`-.__.-'_.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:       `-._    `-.__.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:           `-._        _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:               `-.__.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:
Sep 12 21:17:29 172.17.0.40 redis.0[1]:                 _._
Sep 12 21:17:29 172.17.0.40 redis.0[1]:            _.-``__ ''-._
Sep 12 21:17:29 172.17.0.40 redis.0[1]:       _.-``    `.  `_.  ''-._           Redis 2.8.14 (00000000/0) 64 bit
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   .-`` .-```.  ```\/    _.,_ ''-._
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  (    '      ,       .-`  | `,    )     Running in stand alone mode
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |    `-._   `._    /     _.-'    |     PID: 1
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   `-._    `-._  `-./  _.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |`-._`-._    `-.__.-'    _.-'_.-'|
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |    `-._`-._        _.-'_.-'    |           http://redis.io
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   `-._    `-._`-.__.-'_.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |`-._`-._    `-.__.-'    _.-'_.-'|
Sep 12 21:17:29 172.17.0.40 redis.0[1]:  |    `-._`-._        _.-'_.-'    |
Sep 12 21:17:29 172.17.0.40 redis.0[1]:   `-._    `-._`-.__.-'_.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:       `-._    `-.__.-'    _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:           `-._        _.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:               `-.__.-'
Sep 12 21:17:29 172.17.0.40 redis.0[1]:
Sep 12 21:17:29 172.17.0.40 redis.0[1]: [1] 12 Sep 21:17:29.820 # Server started, Redis version 2.8.14
Sep 12 21:17:29 172.17.0.40 redis.0[1]: [1] 12 Sep 21:17:29.820 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Sep 12 21:17:29 172.17.0.40 redis.0[1]: [1] 12 Sep 21:17:29.820 * The server is now ready to accept connections on port 6379
Sep 12 21:17:29 172.17.0.40 redis.0[1]: [1] 12 Sep 21:17:29.820 # Server started, Redis version 2.8.14
Sep 12 21:17:29 172.17.0.40 redis.0[1]: [1] 12 Sep 21:17:29.820 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Sep 12 21:17:29 172.17.0.40 redis.0[1]: [1] 12 Sep 21:17:29.820 * The server is now ready to accept connections on port 6379

Oddly, when I run without detaching the container:

ubuntu@ip-10-80-72-198:~$ sudo docker run --name redis.1 --rm -p 10000:6379 dockerfile/redis
sudo: unable to resolve host ip-10-80-72-198
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 2.8.14 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in stand alone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

[1] 12 Sep 21:20:04.539 # Server started, Redis version 2.8.14
[1] 12 Sep 21:20:04.540 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[1] 12 Sep 21:20:04.540 * The server is now ready to accept connections on port 6379

I get no logs at all at startup:

ubuntu@ip-10-80-72-198:~$ sudo tail -f /var/log/syslog
...
Sep 12 21:20:04 172.17.0.40 registrator[1]: 2014/09/12 21:20:04 registrator: added: ed54a003e2c5 080a3c722ca2:redis.1:6379

And when I Ctrl+C out of the container and check syslog I see logs but they are duplicated again:

ubuntu@ip-10-80-72-198:~$ sudo tail -f /var/log/syslog
...
Sep 12 21:20:04 172.17.0.40 registrator[1]: 2014/09/12 21:20:04 registrator: added: ed54a003e2c5 080a3c722ca2:redis.0:6379
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1 | signal handler] (1410557604) Received SIGINT scheduling shutdown...
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1 | signal handler] (1410557604) Received SIGINT scheduling shutdown...
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.549 # User requested shutdown...
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.549 * Saving the final RDB snapshot before exiting.
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.549 # User requested shutdown...
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.549 * Saving the final RDB snapshot before exiting.
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.585 * DB saved on disk
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.585 * DB saved on disk
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.585 # Redis is now ready to exit, bye bye...
Sep 12 21:33:24 172.17.0.40 redis.0[1]: [1] 12 Sep 21:33:24.585 # Redis is now ready to exit, bye bye...
Sep 12 21:33:24 172.17.0.40 registrator[1]: 2014/09/12 21:33:24 registrator: removed: ed54a003e2c5 080a3c722ca2:redis.0:6379

Any clues?

Hostname is not correctly set

This is the line reaching logstash the third parameter is indicated to be replaced with the actual hostname but it is not.

<14>2015-02-09T14:25:01Z logspout dev_zservice_1[1]: 2015-02-10 11:55:38.496 INFO 1 --- [tp1302304527-19] c.z.service.DefaultInvoiceService : Creating with DefaultInvoiceService started...

docker exec -ti dev_zservice_1 sh   
/ # env   
HOSTNAME=zservice

support tcp syslog

The only problem is that using the stdlib syslog package prevents you from changing the syslog tag across writes. That's why right now I Dial once for every message sent over syslog (but it's ok because it's udp). However, TCP would require a persistent connection.

Perhaps the solution is to vendor and fork the syslog package as it is not that complicated. A patch upstream seems unlikely to be merged because this is a much more uncommon use case (logging on somebody else's behalf).

Duplicate logs

When I try to use logspout, I see duplicate logs arrive in papertrail, and it seems that the log is truly being sent twice to papertrail (i.e. the problem is not on papertrail's end). Here is my setup:

$ sudo docker -v
Docker version 1.0.1, build 990021a

$ sudo docker run --rm -e "DEBUG=1" -h="logspout-test" -v=/var/run/docker.sock:/tmp/docker.sock progrium/logspout syslog://logs.papertrailapp.com:xxxxx

In another terminal,

$ sudo docker run --rm flynn/busybox echo hello world
hello world

The logspout output:

2014/06/26 21:30:57 attach: 64811a11783c success
2014/06/26 21:30:57 routing all to syslog://logs.papertrailapp.com:xxxxx
2014/06/26 21:30:57 loading and persisting routes in /mnt/routes
2014/06/26 21:30:57 logspout serving http on :8000
2014/06/26 21:31:03 event: 33a4cd018d64 create
2014/06/26 21:31:03 event: 33a4cd018d64 create
2014/06/26 21:31:03 event: 33a4cd018d64 start
2014/06/26 21:31:03 event: 33a4cd018d64 start
2014/06/26 21:31:03 attach: 33a4cd018d64 success
2014/06/26 21:31:03 attach: 33a4cd018d64 success
2014/06/26 21:31:03 attach: 33a4cd018d64 finished
2014/06/26 21:31:03 attach: 33a4cd018d64 finished
2014/06/26 21:31:04 event: 33a4cd018d64 die
2014/06/26 21:31:04 event: 33a4cd018d64 die
2014/06/26 21:31:04 event: 33a4cd018d64 destroy
2014/06/26 21:31:04 event: 33a4cd018d64 destroy

and the corresponding papertrail output:

Jun 26 14:30:56 logspout-test stupefied_engelbart:  2014/06/26 21:30:57 routing all to syslog://logs.papertrailapp.com:xxxxx
Jun 26 14:30:56 logspout-test stupefied_engelbart:  2014/06/26 21:30:57 loading and persisting routes in /mnt/routes 
Jun 26 14:30:56 logspout-test stupefied_engelbart:  2014/06/26 21:30:57 logspout serving http on :8000 
Jun 26 14:31:01 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 event: 33a4cd018d64 create 
Jun 26 14:31:01 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 event: 33a4cd018d64 create 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 event: 33a4cd018d64 start 
Jun 26 14:31:02 logspout-test happy_feynman:  hello world 
Jun 26 14:31:02 logspout-test happy_feynman:  hello world 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 event: 33a4cd018d64 start 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 attach: 33a4cd018d64 success 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 attach: 33a4cd018d64 success 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 attach: 33a4cd018d64 finished 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:03 attach: 33a4cd018d64 finished 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:04 event: 33a4cd018d64 die 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:04 event: 33a4cd018d64 die 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:04 event: 33a4cd018d64 destroy 
Jun 26 14:31:02 logspout-test stupefied_engelbart:  2014/06/26 21:31:04 event: 33a4cd018d64 destroy

This is with the most recent progrium/logspout image from the public docker registry. I am able to consistently reproduce this issue in both Ubuntu 14.04 and in boot2docker.

I built my own logspout image with these two lines commented out. This fixes the duplicate logging, but I guess it probably reintroduces the problem with the go-dockerclient bug it's intended to work around.

panic: runtime error: assignment to entry in nil map

Hi @progrium,

When I run the current master pointing to loggly url like that: rfc5424://logs-01.loggly.com:514?structuredData=<token>@41058
I get an exception like that:

2015-04-07T00:48:16.268748415Z # logspout v3-master by gliderlabs
2015-04-07T00:48:16.268825080Z # adapters: tcp syslog raw udp
2015-04-07T00:48:16.268952603Z # options : persist:/mnt/routes
2015-04-07T00:48:16.269574474Z panic: runtime error: assignment to entry in nil map
2015-04-07T00:48:16.269611719Z 
2015-04-07T00:48:16.269611719Z goroutine 16 [running]:
2015-04-07T00:48:16.269611719Z runtime.panic(0x7af000, 0xa1d5d3)
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
2015-04-07T00:48:16.269611719Z github.com/gliderlabs/logspout/router.(*RouteManager).AddFromUri(0xc208024de0, 0x7fff0d7f8d38, 0x5a, 0x0, 0x0)
2015-04-07T00:48:16.269611719Z  /go/src/github.com/gliderlabs/logspout/router/routes.go:102 +0x4b3
2015-04-07T00:48:16.269611719Z github.com/gliderlabs/logspout/router.(*RouteManager).Setup(0xc208024de0, 0x0, 0x0)
2015-04-07T00:48:16.269611719Z  /go/src/github.com/gliderlabs/logspout/router/routes.go:184 +0x1bb
2015-04-07T00:48:16.269611719Z main.main()
2015-04-07T00:48:16.269611719Z  /go/src/github.com/gliderlabs/logspout/logspout.go:39 +0x646
2015-04-07T00:48:16.269611719Z 
2015-04-07T00:48:16.269611719Z goroutine 17 [runnable]:
2015-04-07T00:48:16.269611719Z runtime.MHeap_Scavenger()
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/mheap.c:507
2015-04-07T00:48:16.269611719Z runtime.goexit()
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/proc.c:1445
2015-04-07T00:48:16.269611719Z 
2015-04-07T00:48:16.269611719Z goroutine 18 [runnable]:
2015-04-07T00:48:16.269611719Z bgsweep()
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/mgc0.c:1976
2015-04-07T00:48:16.269611719Z runtime.goexit()
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/proc.c:1445
2015-04-07T00:48:16.269611719Z 
2015-04-07T00:48:16.269611719Z goroutine 19 [runnable]:
2015-04-07T00:48:16.269611719Z runfinq()
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/mgc0.c:2606
2015-04-07T00:48:16.269611719Z runtime.goexit()
2015-04-07T00:48:16.269611719Z  /usr/lib/go/src/pkg/runtime/proc.c:1445

The same url seems to be working in latest image, but in latest once a slightly longer log entry occures, it blows up with write udp 54.236.80.3:514: message too long

Is there an issue or am I doing something wrong?

Thanks,
Evgeny

All log lines are doubled

I'm getting all lines from my worker containers doubled in the logspout output. Both when reading the collected data via curl on port 8000 and also when sent to papertrail.

Host:

CoreOS 494.5.0
Docker version 1.3.3, build 5dc1c5a

Test container

FROM progrium/busybox
MAINTAINER Mats Engstrom <[email protected]>
RUN wget -O - http://nodejs.org/dist/v0.10.33/node-v0.10.33-linux-x64.tar.gz | gunzip | tar -x -C /root/
RUN ln -sv /root/node-v0.10.33-linux-x64/lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm
RUN ln -sv /root/node-v0.10.33-linux-x64/bin/node /usr/bin/node
ADD libstdc++.so.6 /usr/lib64/libstdc++.so.6
ADD test.js /test.js
CMD /usr/bin/node /test.js

The test.js file

var cnt=0; setInterval(function(){console.log("Hello "+cnt++);},1000);

Start of logspout

core@ip-10-150-140-163 ~/Docker-test1 $ docker run -d -p 8000:8000 \
>     -v=/var/run/docker.sock:/tmp/docker.sock \
>     progrium/logspout
acdbfa29e637705509f2f55503c829e96fa22dd6fcb62010cf6253fd499b5d56

Start of the test container

docker run --detach viacard/test1

Result

core@ip-10-150-140-163 ~/Docker-test1 $ curl $(docker port `docker ps -lq` 8000)/logs
  drunk_poincare|[martini] Started GET /logs for 172.17.42.1:46125
   jovial_carson|Hello 0
   jovial_carson|Hello 0
   jovial_carson|Hello 1
   jovial_carson|Hello 1
   jovial_carson|Hello 2
   jovial_carson|Hello 2
   jovial_carson|Hello 3
   jovial_carson|Hello 3
   jovial_carson|Hello 4
   jovial_carson|Hello 4

Persist logspout across reboots

I'm pretty sure this is a newb question. Please feel free to direct me somewhere else to answer my question rather than taking the time if it's not related. Setup works fine and running docker run -v=/var/run/docker.sock:/tmp/docker.sock progrium/logspout syslog://logs.papertrailapp.com:55555 works great, but when I ctrl+C or reboot the system, logging is done. What's the best way to set this up to start when the system starts and continue? Thanks.

DNS resolution

With the latest version of logspout I'm having DNS resolution issues. I can resolve google.com over and over again and it resolves, but when I attempt to lookup a dns entry hosted by amazon it intermittently works. We also have a local DNS server backed by consul and I also see intermittent issues.

I tried both using --net host and normal container networking
docker run -it --net host --entrypoint /bin/sh b4a89d11cec4

/ # nslookup logstash.service.consul
Server:    (null)
Address 1: ::1 ip6-localhost
Address 2: 127.0.0.1 localhost

Name:      logstash.service.consul
Address 1: 10.0.10.166 ip-10-0-10-166.ec2.internal
Address 2: 10.0.20.207 ip-10-0-20-207.ec2.internal
/ # nslookup logstash.service.consul
Server:    (null)
Address 1: ::1 ip6-localhost
Address 2: 127.0.0.1 localhost

nslookup: can't resolve 'logstash.service.consul': Name does not resolve
/ #
/ #
/ # nslookup logstash.service.consul
Server:    (null)
Address 1: ::1 ip6-localhost
Address 2: 127.0.0.1 localhost

nslookup: can't resolve 'logstash.service.consul': Name does not resolve
/ # nslookup logstash.service.consul
Server:    (null)
Address 1: ::1 ip6-localhost
Address 2: 127.0.0.1 localhost

nslookup: can't resolve 'logstash.service.consul': Name does not resolve

Container build does not compile custom modules

The RUN command in the Dockerfile (https://github.com/gliderlabs/logspout/blob/master/Dockerfile#L8) needs to be prefixed with ONBUILD to actually run custom module builds. I'll submit a PR that fixes this.

Change to docker 1.7 events breaks logspout

Due to this bug:

moby/moby#12848 (comment)

I have had to use docker 1.7 in my project, or else face running out of memory rapidly.

While implementing logspout (via the docker container, gliderlabs/logspout:master) with docker 1.7, I noticed that a restart of logspout would cause it to crash repeatedly at this line with containers that weren't listed in docker ps:

https://github.com/gliderlabs/logspout/blob/master/router/pump.go#L124

It took a bit, but finally I tracked it down to this change to the events code in docker 1.7:

https://github.com/docker/docker/blob/master/daemon/events/events.go#L28

Basically it looks like the events endpoint now returns the last 64 events and then starts tailing the events. The logspout daemon was crashing because it was working through the past 64 events, and it hit the event that said that the old logspout daemon had just come up.

I want to go ahead and make an attempt at fixing this issue, but first I figured I'd get some people's take on what the behavior should be.

Crash

I've just had this happen:

$ sudo docker.io -v
Docker version 0.9.1, build 3600720
$ sudo docker.io run -d -e "DEBUG=true" -p 8000:8000 -v=/var/run/docker.sock:/tmp/docker.sock progrium/logspout syslog://logs.papertrailapp.com:55555
219c6567d3e07d55d73f0be91f4df34b36ab0464777da7d7dbf6964456015ec3
$ sudo docker.io run stackbrew/ubuntu:13.10 echo hello world
hello world
$ sudo docker.io ps -a
CONTAINER ID        IMAGE                      COMMAND                CREATED              STATUS              PORTS               NAMES
49c83861e5bb        stackbrew/ubuntu:13.10     echo hello world       50 seconds ago       Exit 0                                  condescending_mclean   
219c6567d3e0        progrium/logspout:latest   /bin/logspout syslog   About a minute ago   Exit 2                                  happy_bardeen          
$ sudo docker.io logs 219
2014/05/25 03:14:47 attach: 219c6567d3e0 success
2014/05/25 03:14:47 routing all to syslog://logs.papertrailapp.com:55555
2014/05/25 03:14:47 loading and persisting routes in /mnt/routes
2014/05/25 03:14:47 logspout serving http on :8000
[martini] Started GET /logs for 172.17.42.1:59760
2014/05/25 03:15:05 event: 49c83861e5bb create
2014/05/25 03:15:05 event: 49c83861e5bb start
2014/05/25 03:15:05 attach: 49c83861e5bb success
2014/05/25 03:15:05 attach: 49c83861e5bb finished
2014/05/25 03:15:05 event: 49c83861e5bb die
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x45e7cd]

goroutine 11 [running]:
runtime.panic(0x69ac20, 0xa01e48)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
log/syslog.(*Writer).Write(0x0, 0xc2100cdf80, 0x31, 0x31, 0x31, ...)
    /usr/local/go/src/pkg/log/syslog/syslog.go:178 +0x5d
io.WriteString(0x7f0056810260, 0x0, 0xc2100b8080, 0x31, 0xe, ...)
    /usr/local/go/src/pkg/io/io.go:272 +0xdf
main.syslogStreamer(0x7fff0b321f24, 0x6, 0x7fff0b321f2d, 0x1c, 0x0, ...)
    /vagrant/logspout/logspout.go:65 +0x224
created by main.func·012
    /vagrant/logspout/routes.go:83 +0xbc

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f005680f290, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc21007ab50, 0x72, 0x7f005680e0c0, 0xb)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc21007ab50, 0xb, 0x7f005680e0c0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc21007aaf0, 0x786310, 0x0, 0x7f005680e0c0, 0xb)
    /usr/local/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210000d88, 0x4755fb, 0x7f005666ed08, 0x4755fb)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(*TCPListener).Accept(0xc210000d88, 0x7f005680fcd8, 0xc210000e50, 0xc210060b80, 0x0)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(*Server).Serve(0xc210078640, 0x7f005680fc00, 0xc210000d88, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1622 +0x91
net/http.(*Server).ListenAndServe(0xc210078640, 0xc210078640, 0x7f005666ee08)
    /usr/local/go/src/pkg/net/http/server.go:1612 +0xa0
net/http.ListenAndServe(0xc210000d60, 0x5, 0x7f005680fb80, 0xc21000a140, 0x4, ...)
    /usr/local/go/src/pkg/net/http/server.go:1677 +0x6d
main.main()
    /vagrant/logspout/logspout.go:220 +0xc58

goroutine 3 [chan receive]:
github.com/fsouza/go-dockerclient.(*Client).hijack(0xc21000a4c0, 0x6f3740, 0x4, 0xc210058480, 0x3a, ...)
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/client.go:227 +0x48a
github.com/fsouza/go-dockerclient.(*Client).AttachToContainer(0xc21000a4c0, 0xc21001f460, 0xc, 0x0, 0x0, ...)
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/container.go:538 +0x1e6
main.func·002()
    /vagrant/logspout/attacher.go:63 +0x133
created by main.(*AttachManager).attach
    /vagrant/logspout/attacher.go:75 +0x271

goroutine 4 [semacquire]:
sync.runtime_Syncsemacquire(0xc21006b1c0)
    /usr/local/go/src/pkg/runtime/sema.goc:257 +0xca
sync.(*Cond).Wait(0xc21006b1b0)
    /usr/local/go/src/pkg/sync/cond.go:62 +0x89
io.(*pipe).read(0xc21006b180, 0xc21007d000, 0x1000, 0x1000, 0x0, ...)
    /usr/local/go/src/pkg/io/pipe.go:52 +0x245
io.(*PipeReader).Read(0xc210000940, 0xc21007d000, 0x1000, 0x1000, 0x426a19, ...)
    /usr/local/go/src/pkg/io/pipe.go:134 +0x5f
bufio.(*Reader).fill(0xc21000bc00)
    /usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(*Reader).ReadSlice(0xc21000bc00, 0x40c70a, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(*Reader).ReadBytes(0xc21000bc00, 0xc21009250a, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/bufio/bufio.go:355 +0xbf
main.func·005(0x6fbfe0, 0x6, 0x7f005680f808, 0xc210000940)
    /vagrant/logspout/attacher.go:168 +0x77
created by main.NewLogPump
    /vagrant/logspout/attacher.go:183 +0x164

goroutine 5 [chan send]:
main.(*LogPump).send(0xc21006f990, 0xc2100b8200)
    /vagrant/logspout/attacher.go:193 +0xbd
main.func·005(0x6fbfc0, 0x6, 0x7f005680f808, 0xc210000958)
    /vagrant/logspout/attacher.go:180 +0x354
created by main.NewLogPump
    /vagrant/logspout/attacher.go:184 +0x19d

goroutine 6 [chan receive]:
main.func·001()
    /vagrant/logspout/attacher.go:34 +0xb9
created by main.NewAttachManager
    /vagrant/logspout/attacher.go:41 +0x2ab

goroutine 7 [select]:
main.(*AttachManager).Listen(0xc21000a4e0, 0x7f0056682ed8, 0xc21000bd20, 0xc21000b600)
    /vagrant/logspout/attacher.go:129 +0x50c
main.func·012()
    /vagrant/logspout/routes.go:84 +0xfa
created by main.(*RouteManager).Add
    /vagrant/logspout/routes.go:85 +0x521

goroutine 8 [runnable]:
net.runtime_pollWait(0x7f005680f338, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc21004fae0, 0x72, 0x7f005680e0c0, 0xb)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc21004fae0, 0xb, 0x7f005680e0c0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).Read(0xc21004fa80, 0xc210081000, 0x8009, 0x8009, 0x0, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(*conn).Read(0xc210000a50, 0xc210081000, 0x8009, 0x8009, 0xc210081008, ...)
    /usr/local/go/src/pkg/net/net.go:122 +0xc5
bufio.(*Reader).Read(0xc21000b5a0, 0xc210081000, 0x8009, 0x8009, 0x8009, ...)
    /usr/local/go/src/pkg/bufio/bufio.go:152 +0x106
github.com/fsouza/go-dockerclient/utils.StdCopy(0x7f005680f7b8, 0xc210000948, 0x7f005680f7b8, 0xc210000960, 0x7f005680f530, ...)
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/utils/stdcopy.go:88 +0x129
github.com/fsouza/go-dockerclient.func·001()
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/client.go:213 +0x123
created by github.com/fsouza/go-dockerclient.(*Client).hijack
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/client.go:216 +0x42c

goroutine 10 [select]:
github.com/fsouza/go-dockerclient.(*eventMonitoringState).monitorEvents(0xa070c0, 0xc21000a4c0)
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/event.go:163 +0x2ff
created by github.com/fsouza/go-dockerclient.(*eventMonitoringState).enableEventMonitoring
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/event.go:136 +0x111

goroutine 13 [IO wait]:
net.runtime_pollWait(0x7f005680f1e8, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc21007ae60, 0x72, 0x7f005680e0c0, 0xb)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc21007ae60, 0xb, 0x7f005680e0c0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).Read(0xc21007ae00, 0xc21008f000, 0x1000, 0x1000, 0x0, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(*conn).Read(0xc210000e10, 0xc21008f000, 0x1000, 0x1000, 0x0, ...)
    /usr/local/go/src/pkg/net/net.go:122 +0xc5
bufio.(*Reader).fill(0xc21000bea0)
    /usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(*Reader).Read(0xc21000bea0, 0xc2100911b5, 0xe4b, 0xe4b, 0xe4b, ...)
    /usr/local/go/src/pkg/bufio/bufio.go:159 +0x1a4
bufio.(*Scanner).Scan(0xc21008d150, 0xc210074c00)
    /usr/local/go/src/pkg/bufio/scan.go:165 +0x487
github.com/fsouza/go-dockerclient.func·004(0x7f005680f530, 0xc21000bea0)
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/event.go:264 +0x152
created by github.com/fsouza/go-dockerclient.(*Client).eventHijack
    /home/vagrant/.go/src/github.com/fsouza/go-dockerclient/event.go:280 +0x4a1

goroutine 15 [select]:
main.(*AttachManager).Listen(0xc21000a4e0, 0x7f005664cb08, 0xc21000b0c0, 0xc21000b120)
    /vagrant/logspout/attacher.go:129 +0x50c
main.func·007(0x7f005680fef8, 0xc210092400, 0xc21006c5b0, 0xc210099090)
    /vagrant/logspout/logspout.go:181 +0x326
reflect.Value.call(0x666380, 0xc21001ee50, 0x130, 0x6f05e0, 0x4, ...)
    /usr/local/go/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x666380, 0xc21001ee50, 0x130, 0xc210078960, 0x3, ...)
    /usr/local/go/src/pkg/reflect/value.go:345 +0x9d
github.com/codegangsta/inject.(*injector).Invoke(0xc21007c3c0, 0x666380, 0xc21001ee50, 0x56f9ae, 0x620fe0, ...)
    /home/vagrant/.go/src/github.com/codegangsta/inject/inject.go:102 +0x304
github.com/go-martini/martini.(*context).Invoke(0xc2100787d0, 0x666380, 0xc21001ee50, 0x6ac100, 0x4ad6a0, ...)
    /home/vagrant/.go/src/github.com/go-martini/martini/env.go:1 +0x4c
github.com/go-martini/martini.(*routeContext).run(0xc2100990c0)
    /home/vagrant/.go/src/github.com/go-martini/martini/router.go:350 +0x74
github.com/go-martini/martini.(*route).Handle(0xc21001feb0, 0x7f005680ff30, 0xc2100787d0, 0x7f005680fef8, 0xc210092400)
    /home/vagrant/.go/src/github.com/go-martini/martini/router.go:229 +0xe2
github.com/go-martini/martini.(*router).Handle(0xc21001fe10, 0x7f005680fef8, 0xc210092400, 0xc21006c5b0, 0x7f005680ff30, ...)
    /home/vagrant/.go/src/github.com/go-martini/martini/router.go:112 +0x16c
github.com/go-martini/martini.Router.Handle·fm(0x7f005680fef8, 0xc210092400, 0xc21006c5b0, 0x7f005680ff30, 0xc2100787d0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:111 +0x60
reflect.Value.call(0x6662e0, 0xc21000a120, 0x130, 0x6f05e0, 0x4, ...)
    /usr/local/go/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x6662e0, 0xc21000a120, 0x130, 0xc210078870, 0x3, ...)
    /usr/local/go/src/pkg/reflect/value.go:345 +0x9d
github.com/codegangsta/inject.(*injector).Invoke(0xc21007c3c0, 0x6662e0, 0xc21000a120, 0xa0fcd8, 0x0, ...)
    /home/vagrant/.go/src/github.com/codegangsta/inject/inject.go:102 +0x304
github.com/go-martini/martini.(*context).run(0xc2100787d0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:165 +0x62
github.com/go-martini/martini.(*context).Next(0xc2100787d0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:156 +0x2b
github.com/go-martini/martini.func·004(0x7f005680ff30, 0xc2100787d0, 0xc21001fe60)
    /home/vagrant/.go/src/github.com/go-martini/martini/recovery.go:140 +0x76
reflect.Value.call(0x644ba0, 0x783340, 0x130, 0x6f05e0, 0x4, ...)
    /usr/local/go/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x644ba0, 0x783340, 0x130, 0xc210038ea0, 0x2, ...)
    /usr/local/go/src/pkg/reflect/value.go:345 +0x9d
github.com/codegangsta/inject.(*injector).Invoke(0xc21007c3c0, 0x644ba0, 0x783340, 0xc21001fe60, 0x7f005680fef8, ...)
    /home/vagrant/.go/src/github.com/codegangsta/inject/inject.go:102 +0x304
github.com/go-martini/martini.(*context).run(0xc2100787d0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:165 +0x62
github.com/go-martini/martini.(*context).Next(0xc2100787d0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:156 +0x2b
github.com/go-martini/martini.func·001(0x7f005680fef8, 0xc210092400, 0xc21006c5b0, 0x7f005680ff30, 0xc2100787d0, ...)
    /home/vagrant/.go/src/github.com/go-martini/martini/logger.go:25 +0x292
reflect.Value.call(0x67f6a0, 0x783330, 0x130, 0x6f05e0, 0x4, ...)
    /usr/local/go/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x67f6a0, 0x783330, 0x130, 0xc21000bf60, 0x4, ...)
    /usr/local/go/src/pkg/reflect/value.go:345 +0x9d
github.com/codegangsta/inject.(*injector).Invoke(0xc21007c3c0, 0x67f6a0, 0x783330, 0xc21007c3c0, 0x7f005680fe40, ...)
    /home/vagrant/.go/src/github.com/codegangsta/inject/inject.go:102 +0x304
github.com/go-martini/martini.(*context).run(0xc2100787d0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:165 +0x62
github.com/go-martini/martini.(*Martini).ServeHTTP(0xc210058680, 0x7f005680fe08, 0xc210079b40, 0xc21006c5b0)
    /home/vagrant/.go/src/github.com/go-martini/martini/martini.go:69 +0x53
net/http.serverHandler.ServeHTTP(0xc210078640, 0x7f005680fe08, 0xc210079b40, 0xc21006c5b0)
    /usr/local/go/src/pkg/net/http/server.go:1597 +0x16e
net/http.(*conn).serve(0xc210060b80)
    /usr/local/go/src/pkg/net/http/server.go:1167 +0x7b7
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1644 +0x28b

goroutine 16 [chan receive]:
main.httpStreamer(0x7f005680fef8, 0xc210092400, 0xc21006c5b0, 0xc21000b0c0, 0x1)
    /vagrant/logspout/logspout.go:98 +0x1ac
created by main.func·007
    /vagrant/logspout/logspout.go:177 +0x397

goroutine 17 [IO wait]:
net.runtime_pollWait(0x7f005680f140, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc21008dfb0, 0x72, 0x7f005680e0c0, 0xb)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc21008dfb0, 0xb, 0x7f005680e0c0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).Read(0xc21008df50, 0xc21009a000, 0x8000, 0x8000, 0x0, ...)
    /usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(*conn).Read(0xc210000e50, 0xc21009a000, 0x8000, 0x8000, 0x8000, ...)
    /usr/local/go/src/pkg/net/net.go:122 +0xc5
io.Copy(0x7f005680f7b8, 0xc210000ed8, 0x7f005680fd38, 0xc210000e50, 0x0, ...)
    /usr/local/go/src/pkg/io/io.go:352 +0x1c8
net/http.func·006()
    /usr/local/go/src/pkg/net/http/server.go:161 +0x68
created by net/http.(*conn).closeNotify
    /usr/local/go/src/pkg/net/http/server.go:167 +0x1f5

goroutine 19 [syscall]:
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394

I also curled for the logs, and the output there was:

$ curl $(sudo docker.io port `sudo docker.io ps -lq` 8000)/logs
   happy_bardeen|[martini] Started GET /logs for 172.17.42.1:59760
   happy_bardeen|2014/05/25 03:15:05 event: 49c83861e5bb create
   happy_bardeen|2014/05/25 03:15:05 event: 49c83861e5bb start
   happy_bardeen|2014/05/25 03:15:05 attach: 49c83861e5bb success
condescending_mclean|hello world
curl: (18) transfer closed with outstanding read data remaining

Question - systemd journal?

I'm trying to capture logs from CentOS 7 docker containers. They are sending data to systemd journal. What is the best way to proceed? Do I need to change the way all these containers do logging, or is there a way to support surfacing these logs via logspout?

I'm open to suggestions. Ultimately I want to ship these logs to Loggly, so I'd probably have to post process the logs somewhere like the docker daemon host to add in the customer token. I'm open to ideas on that too..like doing a little awk or something... Again appriciate any ideas on how to use logspout to accomplish all this.

Using udp for syslog

   image: gliderlabs/logspout
   cpu_shares: 128
   restart: always
   volumes:
    - /var/run/docker.sock:/tmp/docker.sock
   command: syslog://splunk.domain.com:514
   environment:
    - HOSTNAME

Currently this is my config for my syslog logger. Is there a way to specify using UDP instead?

Error handling options

Currently logspout continues running if an adapter error occurs and stops sending logs. A flag that instructs logspout to exit upon such errors would make it possible to trigger the container's restart policy and keep the log stream going.

Routes not working

It appears that routes do not work at all. I've tried restart/rerunning my containers and adding/removing the routes several times. The only way it works is if I log all routes by specifying a syslog server as an arg to the container.

Example route

{
    "source": {
        "name": "myapp",
        "types": ["stdout","stderr"]
    },
    "target": {
        "type": "syslog",
        "addr": "logstash:514"
    }
}

I've tried with/without source types, using prefix and filter instead of name, adding a port to the addr/with no port, and also adding syslog:// to the address.

Any help would be greatly appropriated. It's very hard to test this locally if your also running kibana/elasticsearch in a container with the default to all routes.

hostname not recognized?

In one terminal (specifying the hostname)...

ubuntu@ip-10-80-72-198:~$ sudo docker run --name logspout --rm -v /var/run/docker.sock:/tmp/docker.sock -p 8000:8000 -h ip-10-80-72-198 progrium/logspout syslog://10.80.72.198:514
2014/09/12 20:48:00 routing all to syslog://10.80.72.198:514
2014/09/12 20:48:00 loading and persisting routes in /mnt/routes
2014/09/12 20:48:00 logspout serving http on :8000

In another...

ubuntu@ip-10-80-72-198:~$ tail /var/log/syslog
...
Sep 12 20:56:52 172.17.0.40 logspout[1]: 2014/09/12 20:56:52 routing all to syslog://10.80.72.198:514
Sep 12 20:56:52 172.17.0.40 logspout[1]: 2014/09/12 20:56:52 loading and persisting routes in /mnt/routes
Sep 12 20:56:52 172.17.0.40 logspout[1]: 2014/09/12 20:56:52 logspout serving http on :8000
...

I was under the impression logspout would use the container's hostname as the hostname sent to the remote syslog?

redis as destination

I've been trying to send logs to redis using:

docker run -v=/var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME -p 8000:8000 gliderlabs/logspout redis://redis.host.com:6379

Any time frame on when having redis as a destination will be included?

dial unix /tmp/docker.sock: connection refused

I'm trying to run logspout on RHEL and getting the following error:

$ sudo docker run -it --rm -v /var/run:/tmp --name logspout progrium/logspout syslog://example.org:5000
2014/12/01 11:26:40 attacher: dial unix /tmp/docker.sock: connection refused

Support HTTP streamer for Sumo Logic

Sumo Logic has a "Hosted Collector" but its via HTTP and not Syslog:
https://service.sumologic.com/help/Setting_up_a_Hosted_Collector.htm

If its something that Logspout team would consider as PR for the project I wouldn't mind writing the code for it (just need a little direction):

We have syslog streamers defined here:
https://github.com/gliderlabs/logspout/blob/master/router/streamers.go#L43-L54

However, then there is an Exported HttpStreamer but that seems to be writing to HTTP Response object and not a remote HTTP endpoint.
https://github.com/gliderlabs/logspout/blob/master/router/streamers.go#L116

A new httpStreamer function might be a bit confusing (as far as naming is concerned). Any suggestions?

multi-line logs (i.e. stack trace)

When logging stack traces or messages with \n character the lines are appearing in loggly as separate logs.

I'm considering switching my logs to json which would solve this problem, but before doing that I thought I would see if anybody has another solution.

Logspout crashes when route uri contains non-default query parameters

$ docker run -v=/var/run/docker.sock:/var/run/docker.sock gliderlabs/logspout:master syslog://xxx/?hello=kitty
# logspout v3-master by gliderlabs
# adapters: raw udp tcp syslog
# options : persist:/mnt/routes
panic: runtime error: assignment to entry in nil map

goroutine 16 [running]:
runtime.panic(0x7af1c0, 0xa1e5d3)
    /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/gliderlabs/logspout/router.(*RouteManager).AddFromUri(0xc208010d50, 0x7fff3c88ff6d, 0x19, 0x0, 0x0)
    /go/src/github.com/gliderlabs/logspout/router/routes.go:102 +0x4b3
github.com/gliderlabs/logspout/router.(*RouteManager).Setup(0xc208010d50, 0x0, 0x0)
    /go/src/github.com/gliderlabs/logspout/router/routes.go:184 +0x1bb
main.main()
    /go/src/github.com/gliderlabs/logspout/logspout.go:39 +0x646

goroutine 19 [finalizer wait]:
runtime.park(0x414ad0, 0xa22c58, 0xa20ea9)
    /usr/lib/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0xa22c58, 0xa20ea9)
    /usr/lib/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/lib/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/lib/go/src/pkg/runtime/proc.c:1445

Ran into this while working on an HTTP adapter. This adapter will buffer messages to not to make too many roundtrips. I was hoping that I could expose the configuration of the buffer (size, timeout) via options (like the filter.id, ... stuff). when trying, i ran into the crash above.

The fix appears to be simple to me (but i am not a really very experienced when it comes to go...) - in routes.go, change line 83 from this:

    r := &Route{
        Address: u.Host,
        Adapter: u.Scheme,
    }

to this:

    r := &Route{
        Address: u.Host,
        Adapter: u.Scheme,
        Options: make(map[string]string),
    }

This works for me, but i am not sure if this is the best thing to do. If it is, happy to make a PR.

Write to files

Any plans for an adapter which writes the routed logs into their own files under a volume? Perhaps the logs could be named based on container name/id fromwhich they're routed.

This would be like a dumbed-down built-in version of running a syslogd container just to collect and write the logs.

It's a mostly stateless log appliance. It's not meant for managing log files or looking at history. It is just a means to get your logs out to live somewhere else, where they belong.

Feel free to close if this is out of scope. Acknowledging that this would be somewhat contrary to the above README statement.

attacher: dial unix /tmp/docker.sock: no such file or directory

docker run -d -v /var/run/docker.sock:/var/run/docker.sock gliderlabs/logspout syslog://$MYSERVER:3030

2015/04/09 22:25:16 attacher: dial unix /tmp/docker.sock: no such file or directory

Ignore env variable for containers not working

I've started logspout with the following command:
docker run -d --name logspout --volume=/var/run/docker.sock:/tmp/docker.sock -d -h hostname gliderlabs/logspout syslog://[REDACTED]

And started another container with
docker run -d --name ambassador --expose 1234 -e "LOGSPOUT=ignore" -e svendowideit/ambassador

And I'm still seeing the logs from this container getting pumped into my central logging server. Does order matter wrt which order I start the containers in? I've tried both and am getting the same result, but thought I'd make sure.

Output from docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

incorrect container ID being sent to syslog?

When I am inspecting the logs, I notice that I'm seeing the container ID for logspout instead of the container I'm attaching to. To demonstrate:

root@829e030b5ddc:/var/log/deis# cat deis-cache.log 
2014-07-31 18:30:51  2014-07-31T18:30:51Z aa2e7309f8e4 deis-cache[1]: [35] 31 Jul 18:30:51.066 * 10 changes in 300 seconds. Saving...

Notice that the container ID is aa2e7309f8e4. A print of docker ps shows

core@deis-1 ~ $ docker ps
CONTAINER ID        IMAGE                                      COMMAND                CREATED             STATUS              PORTS                                        NAMES
829e030b5ddc        deis/controller:latest                     /app/bin/boot          28 minutes ago      Up 28 minutes       0.0.0.0:8000->8000/tcp                       deis-controller            
5a87b6ddbcfc        deis/logger:latest                         /app/bin/boot          28 minutes ago      Up 28 minutes       514/tcp, 0.0.0.0:514->514/udp                deis-logger                
5851dabfc344        172.17.8.100:5000/humble-fireball:latest   /runner/init start w   29 minutes ago      Up 29 minutes       0.0.0.0:49158->5000/tcp                      humble-fireball_v2.web.2   
6bacd43f9d6e        172.17.8.100:5000/humble-fireball:latest   /runner/init start w   29 minutes ago      Up 29 minutes       0.0.0.0:49157->5000/tcp                      humble-fireball_v2.web.3   
985c80d66f79        172.17.8.100:5000/humble-fireball:latest   /runner/init start w   2 hours ago         Up 2 hours          0.0.0.0:49156->5000/tcp                      humble-fireball_v2.web.1   
aa2e7309f8e4        progrium/logspout:latest                   /bin/logspout syslog   2 hours ago         Up 2 hours          0.0.0.0:8080->8000/tcp                       boring_hypatia             
6983983c8e6f        deis/builder:latest                        /app/bin/entry /app/   2 hours ago         Up 2 hours          0.0.0.0:2223->22/tcp                         deis-builder               
d78de436ada4        deis/registry:latest                       /app/bin/boot          2 hours ago         Up 2 hours          0.0.0.0:5000->5000/tcp                       deis-registry              
fd7147a09fbb        deis/database:latest                       /app/bin/boot          2 hours ago         Up 2 hours          0.0.0.0:5432->5432/tcp                       deis-database              
fa91d18852b9        deis/cache:latest                          /app/bin/boot          2 hours ago         Up 2 hours          0.0.0.0:6379->6379/tcp                       deis-cache                 
414edcaf909d        deis/router:latest                         /app/bin/boot          2 hours ago         Up 2 hours          0.0.0.0:80->80/tcp, 0.0.0.0:2222->2222/tcp   deis-router

Therefore the message is displaying the logspout container ID instead of deis-cache's container ID. Is this intentional?

Running on CoreOS with docker run -d -p 8080:8000 -v /var/run/docker.sock:/tmp/docker.sock progrium/logspout syslog://172.17.8.100:514 (routable internal IP address for deis-logger)

Works with Docker 1.6?

I could swear that with docker 1.5 everything was working as expected, but now with 1.6, it does not seem to send host and app over the syslog protocol:

docker run -d -v /var/run/docker.sock:/tmp/docker.sock --name logger -h s2 gliderlabs/logspout syslog://MYSERVER:514

I would expect to see "s2" as the host, and the hostname of a container as the app on the MYSERVER side. I believe I was in 1.5, but now both are undefined or blank.

MYSERVER is using node glossy to parse messages on a udp socket, which should handle any RFC format. I have a test-producer that uses node syslogger, which is configured for RFC5424 and it "does the right thing"

2015/03/07 01:19:08 rfc5424: write udp 54.236.80.3:514: message too long

When this occurs, Logspout crashes. Minimally, this should be handled a bit more gracefully so that a badly behaved app can't interfere with log capture.

That said, it looks like the Deis folks have a patch:

https://github.com/deis/deis/pull/2375/files

Document port changed from 8000 to 80

The current docs have various port commands etc that point to port 8000, but this seems to have changed to port 80, on master at least. Could this be noted in the docs?

An exclude option supports

If I have some containers which is I don't want to grab logs and routes into somewhere because they have their own log sending mechanism, having a simple exclude option would be nice. It could take a regexp pattern to match each container name or something, like -e EXCLUDE_PATTERN=[(mongod)|(redis)]...

tag docker images

It would be helpful to tag the docker images (maybe with the first 6 characters of the git sha) or a version if you are versioning.

This would allow production users to pin to a specific working release and avoid having the container switch to a possible non-working one if using "latest".

TCP Loggly example

I am wondering if it is possible to take the log output and forward it over tcp and if so could you provide an example? I am trying do an integration with Loggly.

Non-recoverable error to papertrail

$ docker -v
Docker version 1.6.0, build 4749651

$ docker run --name="logspout" --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/logspout syslog://logs2.papertrailapp.com:<MY_PORT>
2015/05/07 21:50:36 routing all to syslog://logs2.papertrailapp.com:36352
2015/05/07 21:50:36 loading and persisting routes in /mnt/routes
2015/05/07 21:50:36 logspout v2 serving http on :8000
2015/05/07 21:50:39 syslog: dial udp: lookup logs2.papertrailapp.com: Non-recoverable error

Configuring syslog address via environment or configuration file

We use docker-compose, therefore the CMD for logspout is embedded in docker-compose.yml. However, the syslog addresses we use will be different per environment.

Can the syslog URI be provided at runtime in any way other than as a command line argument?

I'm also open to any other suggestions for how to have different destinations for production, staging, test, dev, etc.

PR #88 appears to have broken something in master

I noticed that after pulling master the HTTP adapter I am working on wasn't forwarding messages anymore. Thought this might be an issue on my end but couldn't figure it out. Finally, I tried with stock Logspout against Papertrail, and also, no logs appear in my account.

To reproduce locally, you can open a local TCP listener with Netcat in one terminal:

$ netcat -l 0.0.0.0 2115

Then cd to wherever you have the Logspout source code, master branch, latest (d428516), and start Logspout:

$ DEBUG=1 ROUTE=syslog+tcp://$HOSTIP:2115 make dev

$HOSTIP is the IP of my host.

Then create some messages in a container in yet another window - I use this:

$ docker run --rm --name test ubuntu bash -c 'NEW_UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1); for i in `seq 1 67`; do echo $NEW_UUID Hello $i; sleep .5; done'

I am expecting to see the test messages show up in the window running Netcat, but they don't.

However, if I roll master back to before #88 and then run Logspout again:

$ git checkout 415c0f15c790ad21515f41abe8f79a1acddadc3a
$ DEBUG=1 ROUTE=syslog+tcp://$HOSTIP:2115 make dev

And if I then run the message-generating container again, I see the messages I expect in the Netcat window.

When I try to same with Papertrail, things start showing up there too.

I really don't want to start a wild goose chase, but is anybody else able to reproduce this? I looked the commits in @josharian PR and rolled forward one by one, and at least for me, commit 07555c5 appears to be the culprit. Up to that point in the history of master, I get messages in the Netcat window, and as soon as I apply that commit, I suddenly don't anymore.

Looking at 07555c5 it is not at all clear to me why this is an issue. But when I change this back to wrap the call to log.Fatalf in a func() everything works again. Looking at https://golang.org/ref/spec#Go_statements this change should have been safe...

[question] modifying the log output?

The log output looks a little messy from what our logs currently look. On our syslog server, these are the log entries we are receiving from logspout:

2014-07-31 18:30:51  2014-07-31T18:30:51Z aa2e7309f8e4 humble-fireball_v2.web.2[1]: [...]

We're retrieving some pretty redundant pieces of data here:

The timestamp is duplicated
The container ID from logspout is not required

For reference, this is how we'd like our logs to look like:

2014-07-31 18:30:51 humble-fireball_v2.web.1[1]: [...]

This removes the redundant timestamp and the container ID of logspout. In our use case, users are not concerned where the logs came from. Is there a way we can modify the log format sent to an external logger? I'd be happy to take a stab at making this extensible if it's possible. :)

slice bounds out of range at attacher.go:35

I see the following error/crash in my logs -- not sure what could have caused it.

runtime error: slice bounds out of range

goroutine 133 [running]:
runtime.panic(0x717920, 0x91fdaf)
    /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
main.func·001()
    /home/core/gopath/src/github.com/bountylabs/logspout/attacher.go:35 +0x370
created by main.NewAttachManager
     /home/core/gopath/src/github.com/bountylabs/logspout/attacher.go:41 +0x2bb

I'm using the logspout binary from the master branch, with md5sum 973c2c676255c06f2ea82b196b4af925, running on CentOS 6 + Docker 1.4.1.

The complete log is at http://pastebin.com/Crt8C2y1

Allow structured data

Allow the inclusion of structured header data to be compatible with packages such as loggly that require auth tokens and similar inside the syslog header:

see https://www.loggly.com/docs/streaming-syslog-without-using-files/ for an example of headers

Option to specify program name per container

It'd be awesome if there were a way to specify the program name per docker container via environment variables inside the container (similar to how you can set the LOGSPOUT=ignore variable inside a container) that tells logspout what name to give it when routing the log.

Use cases:

If you're using a container orchestration service (Tutum, Flynn, Dokku) in some cases you may have no control over the --name option of your containers when they're deployed. For example, Tutum assigns a GUID to the name field of each container you deploy. So when using logspout my logs in papertrail end up looking like this:

Note the program labels such as f0d9d99a-18e0-4097-91fd-1e496a451cd0 which are automatically assigned by Tutum aren't helpful, since they have no semantic meaning and change on each deploy. If I could specify an variable LOGSPOUT_NAME=my-containers-app in the container and have logspout use that instead then it would be much easier to filter logs this way.

Even if you do have control over container names, you might want to group them in a custom way. If you have multiple instances of the same service running each container has to have a unique name. So you might end up giving the names app-container-1, app-container-2 in Docker, but want to have all the logs appear under the name app-container for simplicity.

Not sure if it's possible, but I think it'd be a very useful feature to have.