cron not started and as a result logrotate does not run about syslog-ng-docker HOT 11 CLOSED

Just an FYI, I forked the project and am implementing/testing these changes. I have some time so though I would fix the issue. :)

from syslog-ng-docker.

Thanks for the idea.

The alternative would be to use the syslog-ng image as a base image, and build an improved version with it, according to your needs.

Another alternative would be that syslog-ng logs to a volume. And another logrotate specific image mounts the volume and does the logrotate? The two containers need to share pid namespace, so that logrotate could reload/reopen syslog-ng, so this can be complex, I did not think this entirely through. But it may work.

I would only put new components into the base image, only if the majority of users who need syslog-ng will need the component too. Users often use the date macros for log rotating, hence it is not a necessary thing. It cannot replace logrotate (no compression support, no cleanup), but for simpler cases it is good enough.

On the other hand, it would increase docker complexity. Now the container should run two processes. What is the best way to do that? Run the two processes with systemd? Or supervisord? There are alternatives or preferences. Which cron implementation to use? The builtin? This might be again question of preference.

However, I am well interested in your experience and the result. If you managed to pull this together, I think it would be well worth documenting it. If it is short guide with a few steps, you could add it to the readme file. If it is more complex, then I think a new wiki entry would be nice.

from syslog-ng-docker.

So, Thinking this through you did bring up some good points. Using another container to rotate the logs may work in a stand alone docker provided you had an orchestrator manage restarting the syslog-ng image but I dont really see it working well. Just due to the interactions and the needed communications between the containers.

As to adding new components, well both cron and logrotate are already in the syslog-ng container image so you are not really adding as much as you are activating the existing components. Though you do bring up a point I had not thought about. Being that not everyone would want to rotate the logs. Im not sure why someone would not want to rotate the logs, but I am sure there is a use case.

So, in the fork I created It is working. I use a start script to first bring up the cron service before starting syslog-ng. To also start cron/logrotate you would set an environment variable LOGROTATE=yes if that environment variable is not set it will not start cron.

The default logrotate for syslog-ng rotates everything ending with a .log but you can attach your own logrotate config to /etc/logrotate.d/syslog-ng

So a docker command to start would look something like this.

sudo docker run -it -v "$PWD/syslog-ng.conf":/etc/syslog-ng/syslog-ng.conf -v "%PWD/syslog-ng-rotate":/etc/logrotate.d/syslog-ng -e LOGROTATE=yes balabit/syslog-ng:latest

I could include a swarm yaml file to bring it up in a swarm as well.

from syslog-ng-docker.

Unfortunately, for me it does not work. Here is what I tried:

Building a test image called test.

docker build . --tag test

I edited the syslog-ng config to:

$ cat syslog-ng.conf
@version: 3.26
@include "scl.conf"

log {
    source {
      internal();
      example-msg-generator();
    };
    destination { file("/tmp/test/output.txt"); };
};

My goal is to logrotate output.txt.

I edited the logrotate file:

$ cat syslog-ng
/tmp/test/output.txt
{
        rotate 7
        size 1K
        missingok
        notifempty
        postrotate
                syslog-ng-ctl reload > /dev/null
        endscript
}
$ ls -l syslog-ng
-rw-rw-r-- 1 furiel furiel 181 ápr   20 17:20 syslog-ng

Started the container with:

docker run --name test --rm -ti -v `pwd`/output:/tmp/test -v `pwd`/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf -v `pwd`/syslog-ng:/etc/logrotate.d/syslog-ng -e LOGROTATE=yes test

Looking in the output directory, I can see output.txt is not rotated.

I entered the container to start logrotate manually:

$ docker exec -ti test bash
root@f3033fef82b3:/# logrotate -v -f /etc/logrotate.d/syslog-ng
error: Ignoring /etc/logrotate.d/syslog-ng because of bad file mode - must be 0644 or 0444.
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries

Handling 0 logs

So apparently, some permissions enforced. I rolled 0644 with a chmod.

After trying again, still not working. Trying to execute logrotate manually again yielded:

$ docker exec -ti test bash
root@e4cb081ffb18:/# logrotate -v -f /etc/logrotate.d/syslog-ng
Ignoring /etc/logrotate.d/syslog-ng because the file owner is wrong (should be root or user with uid 0).
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries

Handling 0 logs

So this means the owner should be root. I could fix that as well with chown.

After trying again, still no automatic logrotate. But this time, if I entered to the container, and started manually: it works.

$ docker exec -ti test bash
root@8616961616cc:/# logrotate -v -f /etc/logrotate.d/syslog-ng
reading config file /etc/logrotate.d/syslog-ng
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries

Handling 1 logs

rotating pattern: /tmp/test/output.txt
 forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /tmp/test/output.txt
Creating new state
  Now: 2020-04-20 15:30
  Last rotated at 2020-04-20 15:00
  log needs rotating
rotating log /tmp/test/output.txt, log->rotateCount is 7
dateext suffix '-20200420'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /tmp/test/output.txt.7 to /tmp/test/output.txt.8 (rotatecount 7, logstart 1, i 7),
old log /tmp/test/output.txt.7 does not exist
renaming /tmp/test/output.txt.6 to /tmp/test/output.txt.7 (rotatecount 7, logstart 1, i 6),
old log /tmp/test/output.txt.6 does not exist
renaming /tmp/test/output.txt.5 to /tmp/test/output.txt.6 (rotatecount 7, logstart 1, i 5),
old log /tmp/test/output.txt.5 does not exist
renaming /tmp/test/output.txt.4 to /tmp/test/output.txt.5 (rotatecount 7, logstart 1, i 4),
old log /tmp/test/output.txt.4 does not exist
renaming /tmp/test/output.txt.3 to /tmp/test/output.txt.4 (rotatecount 7, logstart 1, i 3),
old log /tmp/test/output.txt.3 does not exist
renaming /tmp/test/output.txt.2 to /tmp/test/output.txt.3 (rotatecount 7, logstart 1, i 2),
old log /tmp/test/output.txt.2 does not exist
renaming /tmp/test/output.txt.1 to /tmp/test/output.txt.2 (rotatecount 7, logstart 1, i 1),
old log /tmp/test/output.txt.1 does not exist
renaming /tmp/test/output.txt.0 to /tmp/test/output.txt.1 (rotatecount 7, logstart 1, i 0),
old log /tmp/test/output.txt.0 does not exist
log /tmp/test/output.txt.8 doesn't exist -- won't try to dispose of it
renaming /tmp/test/output.txt to /tmp/test/output.txt.1
running postrotate script

So with this final version, manual logrotate works, but automatic still does not.
If I look for the cron process on the host, I find two of them as expected: one should be for my host, and one should run in the container. So the service is indeed started. It is strange why it does not work.

$ ps aux | grep cron
root      1087  0.0  0.0  30552  3276 ?        Ss   08:55   0:00 /usr/sbin/cron -f
root     29591  0.0  0.0  27996  2264 ?        Ss   17:29   0:00 /usr/sbin/cron
furiel   30177  0.0  0.0  15772  1028 pts/11   S+   17:33   0:00 grep --color=auto cron
$

from syslog-ng-docker.

Oh I think I know what I missed. logrotate is executed through cron.

logrotate is executed daily:

root@0de32d904874:/etc# file /etc/cron.daily/logrotate 
/etc/cron.daily/logrotate: POSIX shell script, ASCII text executable

So probably I should have waited one day to see the automatic logrotate.

from syslog-ng-docker.

Your right, it does run daily.

As to the config permissions, I did not run into that as the first test the Dockerfile added it with the correct permissions and in the swarm I put the config in docker config (docker config create) which is not writable.

As to when it is run, yes it is daily. That is the default setting inside the container. I could set an environment var that lets you set it to hourly, daily, weekley, monthly. That would be easy to do. That would just place the logrotate cronjob into the /etc/cron.(hourly, daily, weekley, monthly)

from syslog-ng-docker.

Ok, added a ROTATE_WHEN env to allow you to set logrotate to run hourly/daily/weekly/monthly in the /etc/cron.(hourly/daily/weekly/monthly)

-e LOGROTATE=yes -e ROTATE_WHEN=daily

from syslog-ng-docker.

I still think this is not the right direction. Resolving these kind of problems is not the responsibility of the base image.

There are a lot of deployment and application specific details. Apart from file owners, permissions, we now need to configure both logrotate and cron, besides syslog-ng. Adding more and more functionality controlled by environment variables increases complexity: inconvenient from deployment point if view, and will become harder to maintain the code as more options are added.

One can hide all these complexities in their own docker files built upon a base image.

from syslog-ng-docker.

If the intended use is as a base image then you should change the main README as it indicates the intended use as "It is intended to be used as a system logging service."

If you intend it to be a system logging service then you need to have a way to rotate those logs.

from syslog-ng-docker.

Bumping for interest, we had a machine fill up recently because the syslog.log files aren't being rotated on a schedule.

Edit:

If anyone has a similar issue and needs logrotate to work with the syslog-ng container I created my own logrotate config that restarts the container when logs are rotated with this snippet.

**DIRECTORY** is a placeholder, this directory is bind mounted from the container to the underlying OS.
**OMMITTED** is the additional settings for the logrotate config
**CONTAINER_NAME** is the name or id of the container, the postrotate docker restart line will release the hold the container has on the syslog.log files.

**DIRECTORY**
{
        **OMITTED**
	postrotate
		docker restart **CONTAINER_NAME**
	endscript
         **OMITTED**
}

from syslog-ng-docker.

Closing, as we are about to archive this repository.
https://github.com/balabit/syslog-ng-docker#deprecation-warning

Please open a new issue in the syslog-ng/syslog-ng repo if you'd like to continue the discussion about log rotation.

from syslog-ng-docker.