Log files are consuming too much disk space. about devops HOT 6 OPEN

@bencourliss and @naphelps Can we get someone to look at this? This issue is causing the community lab exchange to go into restart loops when the disk fills up. Can we provide a mechanism (flag, env var, whatever) that allows us to prune, limit or suppress the log files?

from devops.

@joewxboy @loudonlune

The log level in the Exchange is configurable in the config.json. It can even be turned off if you like.

https://github.com/open-horizon/exchange-api/tree/master/docs#apilogging
Values: https://logback.qos.ch/apidocs/ch/qos/logback/classic/Level.html

from devops.

INFO is the config default.

from devops.

@joewxboy @loudonlune

It's my understanding that all of the containers write to stdout correct? If so then is that captured in /var/log/messages or /var/log/syslog? You should be able to configure your system with logrotate to make sure those files stay manageable. Outside of controlling what gets logged as Nathan mentions, I don't think this is something that should be configured in the all-in-one mgmt hub script.

from devops.

@bencourliss The containers are writing their logs in /var/log/docker/containers/*/*-json.log as a list of serialized JSON objects, where * represents any container's ID. Those are the log files shown in the screenshot that are using high amounts of disk space.

All of the containers write to stdout inside the container environment. Docker reads that and uses a "logging driver" to choose what happens to that output. There is a driver that writes to the syslog, but that isn't the default behavior. The default logging driver is the json-file driver, which is what's creating these large files. There are other drivers that are lighter on disk space that you can choose from here: https://docs.docker.com/config/containers/logging/configure/

I can set the default logging driver in the daemon config to fix this for now, but whether this should be the default behavior of the management hub after it's installed out-of-the-box is the real question.

from devops.

@loudonlune ah, got it. It still seems to me to be something that should be configured on the system itself. I don't feel the management hub should be configuring the system-wide Docker daemon settings. That seems to be more suited for the system administrator. Perhaps it would be good to document it. https://docs.docker.com/config/containers/logging/configure/

from devops.