Comments (11)
Just an FYI, I forked the project and am implementing/testing these changes. I have some time so though I would fix the issue. :)
from syslog-ng-docker.
Thanks for the idea.
The alternative would be to use the syslog-ng image as a base image, and build an improved version with it, according to your needs.
Another alternative would be that syslog-ng logs to a volume. And another logrotate specific image mounts the volume and does the logrotate? The two containers need to share pid namespace, so that logrotate could reload/reopen syslog-ng, so this can be complex, I did not think this entirely through. But it may work.
I would only put new components into the base image, only if the majority of users who need syslog-ng will need the component too. Users often use the date macros for log rotating, hence it is not a necessary thing. It cannot replace logrotate (no compression support, no cleanup), but for simpler cases it is good enough.
On the other hand, it would increase docker complexity. Now the container should run two processes. What is the best way to do that? Run the two processes with systemd? Or supervisord? There are alternatives or preferences. Which cron implementation to use? The builtin? This might be again question of preference.
However, I am well interested in your experience and the result. If you managed to pull this together, I think it would be well worth documenting it. If it is short guide with a few steps, you could add it to the readme file. If it is more complex, then I think a new wiki entry would be nice.
from syslog-ng-docker.
So, Thinking this through you did bring up some good points. Using another container to rotate the logs may work in a stand alone docker provided you had an orchestrator manage restarting the syslog-ng image but I dont really see it working well. Just due to the interactions and the needed communications between the containers.
As to adding new components, well both cron and logrotate are already in the syslog-ng container image so you are not really adding as much as you are activating the existing components. Though you do bring up a point I had not thought about. Being that not everyone would want to rotate the logs. Im not sure why someone would not want to rotate the logs, but I am sure there is a use case.
So, in the fork I created It is working. I use a start script to first bring up the cron service before starting syslog-ng. To also start cron/logrotate you would set an environment variable LOGROTATE=yes if that environment variable is not set it will not start cron.
The default logrotate for syslog-ng rotates everything ending with a .log but you can attach your own logrotate config to /etc/logrotate.d/syslog-ng
So a docker command to start would look something like this.
sudo docker run -it -v "$PWD/syslog-ng.conf":/etc/syslog-ng/syslog-ng.conf -v "%PWD/syslog-ng-rotate":/etc/logrotate.d/syslog-ng -e LOGROTATE=yes balabit/syslog-ng:latest
I could include a swarm yaml file to bring it up in a swarm as well.
from syslog-ng-docker.
Unfortunately, for me it does not work. Here is what I tried:
Building a test image called test
.
docker build . --tag test
I edited the syslog-ng config to:
$ cat syslog-ng.conf
@version: 3.26
@include "scl.conf"
log {
source {
internal();
example-msg-generator();
};
destination { file("/tmp/test/output.txt"); };
};
My goal is to logrotate output.txt
.
I edited the logrotate file:
$ cat syslog-ng
/tmp/test/output.txt
{
rotate 7
size 1K
missingok
notifempty
postrotate
syslog-ng-ctl reload > /dev/null
endscript
}
$ ls -l syslog-ng
-rw-rw-r-- 1 furiel furiel 181 ápr 20 17:20 syslog-ng
Started the container with:
docker run --name test --rm -ti -v `pwd`/output:/tmp/test -v `pwd`/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf -v `pwd`/syslog-ng:/etc/logrotate.d/syslog-ng -e LOGROTATE=yes test
Looking in the output directory, I can see output.txt
is not rotated.
I entered the container to start logrotate manually:
$ docker exec -ti test bash
root@f3033fef82b3:/# logrotate -v -f /etc/logrotate.d/syslog-ng
error: Ignoring /etc/logrotate.d/syslog-ng because of bad file mode - must be 0644 or 0444.
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Handling 0 logs
So apparently, some permissions enforced. I rolled 0644
with a chmod
.
After trying again, still not working. Trying to execute logrotate manually again yielded:
$ docker exec -ti test bash
root@e4cb081ffb18:/# logrotate -v -f /etc/logrotate.d/syslog-ng
Ignoring /etc/logrotate.d/syslog-ng because the file owner is wrong (should be root or user with uid 0).
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Handling 0 logs
So this means the owner should be root. I could fix that as well with chown
.
After trying again, still no automatic logrotate. But this time, if I entered to the container, and started manually: it works.
$ docker exec -ti test bash
root@8616961616cc:/# logrotate -v -f /etc/logrotate.d/syslog-ng
reading config file /etc/logrotate.d/syslog-ng
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Handling 1 logs
rotating pattern: /tmp/test/output.txt
forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /tmp/test/output.txt
Creating new state
Now: 2020-04-20 15:30
Last rotated at 2020-04-20 15:00
log needs rotating
rotating log /tmp/test/output.txt, log->rotateCount is 7
dateext suffix '-20200420'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /tmp/test/output.txt.7 to /tmp/test/output.txt.8 (rotatecount 7, logstart 1, i 7),
old log /tmp/test/output.txt.7 does not exist
renaming /tmp/test/output.txt.6 to /tmp/test/output.txt.7 (rotatecount 7, logstart 1, i 6),
old log /tmp/test/output.txt.6 does not exist
renaming /tmp/test/output.txt.5 to /tmp/test/output.txt.6 (rotatecount 7, logstart 1, i 5),
old log /tmp/test/output.txt.5 does not exist
renaming /tmp/test/output.txt.4 to /tmp/test/output.txt.5 (rotatecount 7, logstart 1, i 4),
old log /tmp/test/output.txt.4 does not exist
renaming /tmp/test/output.txt.3 to /tmp/test/output.txt.4 (rotatecount 7, logstart 1, i 3),
old log /tmp/test/output.txt.3 does not exist
renaming /tmp/test/output.txt.2 to /tmp/test/output.txt.3 (rotatecount 7, logstart 1, i 2),
old log /tmp/test/output.txt.2 does not exist
renaming /tmp/test/output.txt.1 to /tmp/test/output.txt.2 (rotatecount 7, logstart 1, i 1),
old log /tmp/test/output.txt.1 does not exist
renaming /tmp/test/output.txt.0 to /tmp/test/output.txt.1 (rotatecount 7, logstart 1, i 0),
old log /tmp/test/output.txt.0 does not exist
log /tmp/test/output.txt.8 doesn't exist -- won't try to dispose of it
renaming /tmp/test/output.txt to /tmp/test/output.txt.1
running postrotate script
So with this final version, manual logrotate works, but automatic still does not.
If I look for the cron process on the host, I find two of them as expected: one should be for my host, and one should run in the container. So the service is indeed started. It is strange why it does not work.
$ ps aux | grep cron
root 1087 0.0 0.0 30552 3276 ? Ss 08:55 0:00 /usr/sbin/cron -f
root 29591 0.0 0.0 27996 2264 ? Ss 17:29 0:00 /usr/sbin/cron
furiel 30177 0.0 0.0 15772 1028 pts/11 S+ 17:33 0:00 grep --color=auto cron
$
from syslog-ng-docker.
Oh I think I know what I missed. logrotate
is executed through cron
.
logrotate
is executed daily:
root@0de32d904874:/etc# file /etc/cron.daily/logrotate
/etc/cron.daily/logrotate: POSIX shell script, ASCII text executable
So probably I should have waited one day to see the automatic logrotate.
from syslog-ng-docker.
Your right, it does run daily.
As to the config permissions, I did not run into that as the first test the Dockerfile added it with the correct permissions and in the swarm I put the config in docker config (docker config create) which is not writable.
As to when it is run, yes it is daily. That is the default setting inside the container. I could set an environment var that lets you set it to hourly, daily, weekley, monthly. That would be easy to do. That would just place the logrotate cronjob into the /etc/cron.(hourly, daily, weekley, monthly)
from syslog-ng-docker.
Ok, added a ROTATE_WHEN env to allow you to set logrotate to run hourly/daily/weekly/monthly in the /etc/cron.(hourly/daily/weekly/monthly)
-e LOGROTATE=yes -e ROTATE_WHEN=daily
from syslog-ng-docker.
I still think this is not the right direction. Resolving these kind of problems is not the responsibility of the base image.
There are a lot of deployment and application specific details. Apart from file owners, permissions, we now need to configure both logrotate
and cron
, besides syslog-ng
. Adding more and more functionality controlled by environment variables increases complexity: inconvenient from deployment point if view, and will become harder to maintain the code as more options are added.
One can hide all these complexities in their own docker files built upon a base image.
from syslog-ng-docker.
If the intended use is as a base image then you should change the main README as it indicates the intended use as "It is intended to be used as a system logging service."
If you intend it to be a system logging service then you need to have a way to rotate those logs.
from syslog-ng-docker.
Bumping for interest, we had a machine fill up recently because the syslog.log files aren't being rotated on a schedule.
Edit:
If anyone has a similar issue and needs logrotate to work with the syslog-ng container I created my own logrotate config that restarts the container when logs are rotated with this snippet.
**DIRECTORY**
is a placeholder, this directory is bind mounted from the container to the underlying OS.
**OMMITTED**
is the additional settings for the logrotate config
**CONTAINER_NAME**
is the name or id of the container, the postrotate docker restart line will release the hold the container has on the syslog.log files.
**DIRECTORY**
{
**OMITTED**
postrotate
docker restart **CONTAINER_NAME**
endscript
**OMITTED**
}
from syslog-ng-docker.
Closing, as we are about to archive this repository.
https://github.com/balabit/syslog-ng-docker#deprecation-warning
Please open a new issue in the syslog-ng/syslog-ng repo if you'd like to continue the discussion about log rotation.
from syslog-ng-docker.
Related Issues (20)
- Cannot see logs from network source HOT 3
- Setup not reproductible HOT 5
- Doesnt start HOT 7
- Reload syslog-ng in container HOT 4
- Error creating table with pgsql HOT 2
- IP address of ASA HOT 10
- [Feature] Raspberry Pi / ARM Support HOT 5
- Error response when trying to run HOT 2
- Question, what does the -edv at the very end of the docker run command do ? HOT 2
- 'No such file or directory' When using variables in the destination path. HOT 2
- Default log filename? HOT 2
- Unable to initialize database access (DBI); rc=’-1′, error=’No such file or directory (2)’ HOT 2
- syslog-ng reload fails
- syslog-ng reload fails HOT 6
- syslog-ng stops relaying messages over udp after "bad lenght" HOT 4
- How to stop container creating logs/dirs as root? HOT 4
- Change log file name to container name HOT 2
- Suppport for TLSv1.3 HOT 2
- how to enable mqtt as a source HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from syslog-ng-docker.