I am setting up Zabbix server in our environment. Connections from the Zabbix server to non-docker Hosts/servers is working flawlessly. We are deploying the Dockbix agent via Kubernetes, and are unable to get stats from containers, and it is flooding our central logging Server.
During startup of the Dockbix instance, it seems to correctly initialize and find it's configuration.
It creates 6 Agents -
#0 = main process
#1 = Collector
#2-4 = listener
#5 = Active checks
I am finding it difficult to troubleshoot without access to the container via Docker exec, but I have exported the logs from the central logging server and doctored them to remove personally identifying information.
"Dockbix Agent XXL v0.0.1b limited version""," ,2017-06-19T08:59:25.542-0500,http:docker,_json
"{""line"":""Copyright (C) 2014-2017 Jan Garaj - www.monitoringartist.com""," ,2017-06-19T08:59:25.554-0500,http:docker,_json
"{""line"":""Freeware licence - Usage of this binary is restricted to official monitoringartist Docker images only.""," ,2017-06-19T08:59:25.559-0500,http:docker,_json
"{""line"":""Starting Zabbix Agent [Hostname]. Zabbix 3.2.4 Dockbix Agent XXL (2017-03-25) (revision ).""," ,2017-06-19T08:59:25.563-0500,http:docker,_json
"{""line"":""Press Ctrl+C to exit.""," ,2017-06-19T08:59:25.567-0500,http:docker,_json
"{""line"":""""," ,2017-06-19T08:59:25.572-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.556 Starting Zabbix Agent [Hostname]. Zabbix 3.2.4 Dockbix Agent XXL (2017-03-25) (revision ).""," ,2017-06-19T08:59:25.576-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 **** Enabled features ****""," ,2017-06-19T08:59:25.583-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 IPv6 support: YES""," ,2017-06-19T08:59:25.588-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 TLS support: NO""," ,2017-06-19T08:59:25.594-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 **************************""," ,2017-06-19T08:59:25.598-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.557 using configuration file: /etc/zabbix/zabbix_agentd.conf""," ,2017-06-19T08:59:25.604-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.562 loaded modules: zabbix_module_docker.so, zabbix_module_stress.so""," ,2017-06-19T08:59:25.604-0500,http:docker,_json
"{""line"":"" 10315:20170619:135925.563 agent #0 started [main process]""," ,2017-06-19T08:59:25.607-0500,http:docker,_json
"{""line"":"" 10316:20170619:135925.564 agent #1 started [collector]""," ,2017-06-19T08:59:25.609-0500,http:docker,_json
"{""line"":"" 10317:20170619:135925.564 agent #2 started [listener #1]""," ,2017-06-19T08:59:25.619-0500,http:docker,_json
"{""line"":"" 10318:20170619:135925.565 agent #3 started [listener #2]""," ,2017-06-19T08:59:25.627-0500,http:docker,_json
"{""line"":"" 10319:20170619:135925.565 agent #4 started [listener #3]""," ,2017-06-19T08:59:25.631-0500,http:docker,_json
"{""line"":"" 10320:20170619:135925.565 agent #5 started [active checks #1]""," ,2017-06-19T08:59:25.634-0500,http:docker,_json
"{""line"":"" 10320:20170619:135925.568 active check configuration update from [shp-corp-zab.corporate:10051] started to fail (cannot resolve [shp-corp-zab.corporate])""," ,2017-06-19T08:59:25.636-0500,http:docker,_json
"{""line"":"" 10317:20170619:140034.310 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:34.311-0500,http:docker,_json
"{""line"":"" 10317:20170619:140034.311 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:34.343-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.635 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:34.636-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.636 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance-904c-005056bd0e22_25a1f918/memory.stat'""," ,2017-06-19T09:00:34.638-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.653 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:34.653-0500,http:docker,_json
"{""line"":"" 10318:20170619:140034.653 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//$DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:34.656-0500,http:docker,_json
"{""line"":"" 10319:20170619:140035.713 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:35.713-0500,http:docker,_json
"{""line"":"" 10319:20170619:140035.713 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:35.715-0500,http:docker,_json
"{""line"":"" 10318:20170619:140035.727 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:35.727-0500,http:docker,_json
"{""line"":"" 10318:20170619:140035.727 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:35.728-0500,http:docker,_json
"{""line"":"" 10317:20170619:140036.731 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:36.731-0500,http:docker,_json
"{""line"":"" 10317:20170619:140036.731 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:36.733-0500,http:docker,_json
"{""line"":"" 10318:20170619:140036.781 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:36.781-0500,http:docker,_json
"{""line"":"" 10318:20170619:140036.781 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:36.783-0500,http:docker,_json
"{""line"":"" 10319:20170619:140037.840 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:37.841-0500,http:docker,_json
"{""line"":"" 10319:20170619:140037.840 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:37.842-0500,http:docker,_json
"{""line"":"" 10318:20170619:140037.841 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:37.851-0500,http:docker,_json
"{""line"":"" 10318:20170619:140037.841 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:37.855-0500,http:docker,_json
"{""line"":"" 10319:20170619:140038.880 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:38.880-0500,http:docker,_json
"{""line"":"" 10319:20170619:140038.880 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:38.882-0500,http:docker,_json
"{""line"":"" 10317:20170619:140038.880 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:38.883-0500,http:docker,_json
"{""line"":"" 10317:20170619:140038.881 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice//DockerInstance/cpuacct.stat'""," ,2017-06-19T09:00:38.884-0500,http:docker,_json
"{""line"":"" 10318:20170619:140039.891 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:39.891-0500,http:docker,_json
"{""line"":"" 10318:20170619:140039.891 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-/dockbix.scope/memory.stat'""," ,2017-06-19T09:00:39.895-0500,http:docker,_json
"{""line"":"" 10317:20170619:140039.903 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:39.903-0500,http:docker,_json
"{""line"":"" 10317:20170619:140039.903 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:39.907-0500,http:docker,_json
"{""line"":"" 10319:20170619:140040.912 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.913-0500,http:docker,_json
"{""line"":"" 10319:20170619:140040.913 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:40.916-0500,http:docker,_json
"{""line"":"" 10317:20170619:140040.914 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.918-0500,http:docker,_json
"{""line"":"" 10317:20170619:140040.914 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:40.918-0500,http:docker,_json
"{""line"":"" 10317:20170619:140040.952 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.953-0500,http:docker,_json
"{""line"":"" 10318:20170619:140040.954 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:40.956-0500,http:docker,_json
"{""line"":"" 10318:20170619:140040.954 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice//DockerInstance/memory.stat'""," ,2017-06-19T09:00:40.960-0500,http:docker,_json
"{""line"":"" 10317:20170619:140041.928 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:41.928-0500,http:docker,_json
"{""line"":"" 10318:20170619:140041.928 Cannot find the [Id] item in the received JSON object""," ,2017-06-19T09:00:41.932-0500,http:docker,_json