Docker-Monitoring based on Cadvisor, InfluxDB, and Grafana
After publishing the article for Docker Monitoring based on cAdvisor, InfluxDB, and Grafana the request came up to share the Grafana Dashboards that were referenced in the article - Docker Monitoring Article
Please take a look a the Docker-Monitoring-xx.json file to view the Grafana Dashboard's json file and check out the queries folders for screenshots of the queries in each graph.
Also added the docker-compose.yml file to easily stand up the entire Monitoring Stack.
Hey ,can you add for screenshot what you added for axes?
Get the datasource not found error after execute docker-compose up and import docker-monitoring.json
May fixed it .
docker-monitoring.json
line 96
"datasource": "influxdb", => "datasource": "cadvisor",
I want to put this into my swarm cluster (6 nodes in dev).
If I understand well cadvisor shall run as global (on every nodes)?
Then I guess there is some work to configure the dashboard to monitor all nodes.
Cheers!
let's say, i have lot of docker environment which restrict to expose metrics to any port due to security reason except secure link. how i will configure it from Cadvisor end and prometheus side to collect the metrics.
Now it shows Grafana datasource configuration rather than CPU query configuration.
Time to update this stack. It is outdated and needs some TLC
Hello,
thanks for your work!
My case
should be
Can you help to adjust default json to your screenshot manner?
I use debian 8.3 x64,
docker-compose up.
Hi Brian,
I am in the process of creating a blog, see this link.
The docker containers were created from docker-compose.yml and the Grafana graphs have been created from docker-monitoring-0.9.json of your repository.
There you can see that the numbers shown in Grafana do not look trustworthy:
After clicking refresh:
After running a CPU stress test and clicking refresh again:
It looks like the graphs are created from random numbers. Have you ever seen something like this?
When trying to create the solution using the docker-compose.yml file, I get repeated errors like follows:
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
How to reproduce:
Step 1: install docker compose via container (not tested via native installation):
# detect, whether sudo is needed:
sudo echo hallo > /dev/null 2>&1 && SUDO=sudo
# download docker-compose wrapper, if docker-compose:
$SUDO docker-compose --version || \
$SUDO docker --version && \
curl -L https://github.com/docker/compose/releases/download/1.8.1/run.sh | $SUDO tee /usr/local/bin/docker-compose && \
$SUDO chmod +x /usr/local/bin/docker-compose
Step 2: clone repository
git clone https://github.com/vegasbrianc/docker-monitoring && \
cd docker-monitoring
Step 3: start containers via docker-compose (I will add the full log):
$ docker-compose up
Creating dockermonitoring_influxdbData_1
Creating dockermonitoring_influxdb_1
Creating dockermonitoring_grafana_1
Creating dockermonitoring_cadvisor_1
Attaching to dockermonitoring_influxdbData_1, dockermonitoring_influxdb_1, dockermonitoring_cadvisor_1, dockermonitoring_grafana_1
dockermonitoring_influxdbData_1 exited with code 0
influxdb_1 | => Starting InfluxDB in background ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 |
influxdb_1 | 8888888 .d888 888 8888888b. 888888b.
influxdb_1 | 888 d88P" 888 888 "Y88b 888 "88b
influxdb_1 | 888 888 888 888 888 888 .88P
influxdb_1 | 888 88888b. 888888 888 888 888 888 888 888 888 8888888K.
influxdb_1 | 888 888 "88b 888 888 888 888 Y8bd8P' 888 888 888 "Y88b
influxdb_1 | 888 888 888 888 888 888 888 X88K 888 888 888 888
influxdb_1 | 888 888 888 888 888 Y88b 888 .d8""8b. 888 .d88P 888 d88P
influxdb_1 | 8888888 888 888 888 888 "Y88888 888 888 8888888P" 8888888P"
influxdb_1 |
influxdb_1 | 2016/10/24 14:27:05 InfluxDB starting, version 0.9.6.1, branch 0.9.6, commit 6d3a8603cfdaf1a141779ed88b093dcc5c528e5e, built 2015-12-10T23:40:23+0000
influxdb_1 | 2016/10/24 14:27:05 Go version go1.4.2, GOMAXPROCS set to 2
influxdb_1 | 2016/10/24 14:27:05 Using configuration at: /config/config.toml
influxdb_1 | [metastore] 2016/10/24 14:27:05 Using data dir: /data/meta
influxdb_1 | [retention] 2016/10/24 14:27:05 retention policy enforcement terminating
influxdb_1 | [monitor] 2016/10/24 14:27:05 shutting down monitor system
influxdb_1 | [handoff] 2016/10/24 14:27:05 shutting down hh service
influxdb_1 | [subscriber] 2016/10/24 14:27:05 closed service
influxdb_1 | run: open server: open meta store: raft: new bolt store: invalid argument
cadvisor_1 | I1024 14:27:05.779434 1 storagedriver.go:42] Using backend storage type "influxdb"
cadvisor_1 | I1024 14:27:05.779493 1 storagedriver.go:44] Caching stats in memory for 2m0s
cadvisor_1 | I1024 14:27:05.779725 1 manager.go:131] cAdvisor running in container: "/docker/d1e759712b0e0937ca6030faeaf2712de90ad3b53738dd190cd4bca263093a5e"
cadvisor_1 | I1024 14:27:05.813577 1 fs.go:107] Filesystem partitions: map[/dev/dm-0:{mountpoint:/rootfs major:252 minor:0 fsType: blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType: blockSize:0}]
cadvisor_1 | I1024 14:27:05.836704 1 machine.go:50] Couldn't collect info from any of the files in "/rootfs/etc/machine-id,/var/lib/dbus/machine-id"
cadvisor_1 | I1024 14:27:05.836804 1 manager.go:166] Machine: {NumCores:2 CpuFrequency:2592000 MemoryCapacity:1569599488 MachineID: SystemUUID:B63CB367-870F-4E48-917F-7E524C2C67A0 BootID:e225b37a-b8e6-466b-9f67-84b74df8e90c Filesystems:[{Device:/dev/sda1 Capacity:246755328} {Device:/dev/dm-0 Capacity:41092214784}] DiskMap:map[252:0:{Name:dm-0 Major:252 Minor:0 Size:41884319744 Scheduler:none} 252:1:{Name:dm-1 Major:252 Minor:1 Size:805306368 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:42949672960 Scheduler:deadline}] NetworkDevices:[{Name:eth0 MacAddress:08:00:27:c7:ba:b5 Speed:1000 Mtu:1500} {Name:eth1 MacAddress:08:00:27:51:9c:7e Speed:1000 Mtu:1500}] Topology:[{Id:0 Memory:1569599488 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:6291456 Type:Unified Level:3}]} {Id:1 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:6291456 Type:Unified Level:3}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown}
cadvisor_1 | I1024 14:27:05.837622 1 manager.go:172] Version: {KernelVersion:4.2.0-42-generic ContainerOsVersion:Alpine Linux v3.2 DockerVersion:1.12.1 CadvisorVersion:0.20.5 CadvisorRevision:9aa348f}
cadvisor_1 | E1024 14:27:05.851932 1 manager.go:208] Docker container factory registration failed: docker found, but not using native exec driver.
cadvisor_1 | I1024 14:27:05.856577 1 factory.go:94] Registering Raw factory
cadvisor_1 | I1024 14:27:05.899932 1 manager.go:1000] Started watching for new ooms in manager
cadvisor_1 | I1024 14:27:05.900577 1 manager.go:252] Starting recovery of all containers
cadvisor_1 | I1024 14:27:05.910907 1 manager.go:257] Recovery completed
cadvisor_1 | I1024 14:27:05.931810 1 cadvisor.go:106] Starting cAdvisor version: 0.20.5-9aa348f on port 8080
grafana_1 | 2016/10/24 14:27:05 [I] Starting Grafana
grafana_1 | 2016/10/24 14:27:05 [I] Version: 2.6.0, Commit: v2.6.0, Build date: 2015-12-14 14:18:01 +0000 UTC
grafana_1 | 2016/10/24 14:27:05 [I] Configuration Info
grafana_1 | Config files:
grafana_1 | [0]: /usr/share/grafana/conf/defaults.ini
grafana_1 | [1]: /etc/grafana/grafana.ini
grafana_1 | Command lines overrides:
grafana_1 | [0]: default.paths.data=/var/lib/grafana
grafana_1 | [1]: default.paths.logs=/var/log/grafana
grafana_1 | Paths:
grafana_1 | home: /usr/share/grafana
grafana_1 | data: /var/lib/grafana
grafana_1 | logs: /var/log/grafana
grafana_1 |
grafana_1 | 2016/10/24 14:27:05 [I] Database: sqlite3
grafana_1 | 2016/10/24 14:27:05 [I] Migrator: Starting DB migration
grafana_1 | 2016/10/24 14:27:05 [I] Migrator: exec migration id: create migration_log table
grafana_1 | 2016/10/24 14:27:05 [I] Migrator: exec migration id: create user table
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add unique index user.login
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add unique index user.email
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index UQE_user_login - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index UQE_user_email - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Rename table user to user_v1 - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create user table v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_user_login - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_user_email - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: copy data_source v1 to v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Drop old table user_v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create temp user table v1-7
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_temp_user_email - v1-7
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_temp_user_org_id - v1-7
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_temp_user_code - v1-7
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_temp_user_status - v1-7
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create star table
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add unique index star.user_id_dashboard_id
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create org table v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_org_name - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create org_user table v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_org_user_org_id - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_org_user_org_id_user_id - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: copy data account to org
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: skipping migration id: copy data account to org, condition not fulfilled
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: copy data account_user to org_user
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Drop old table account
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Drop old table account_user
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create dashboard table
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add index dashboard.account_id
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add unique index dashboard_account_id_slug
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create dashboard_tag table
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add unique index dashboard_tag.dasboard_id_term
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index UQE_dashboard_tag_dashboard_id_term - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Rename table dashboard to dashboard_v1 - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create dashboard v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_dashboard_org_id - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_dashboard_org_id_slug - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: copy dashboard v1 to v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop table dashboard_v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: alter dashboard.data to mediumtext v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create data_source table
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add index data_source.account_id
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add unique index data_source.account_id_name
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index IDX_data_source_account_id - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index UQE_data_source_account_id_name - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Rename table data_source to data_source_v1 - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create data_source table v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_data_source_org_id - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_data_source_org_id_name - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: copy data_source v1 to v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Drop old table data_source_v1 #2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Add column with_credentials
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create api_key table
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add index api_key.account_id
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add index api_key.key
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: add index api_key.account_id_name
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index IDX_api_key_account_id - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index UQE_api_key_key - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop index UQE_api_key_account_id_name - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Rename table api_key to api_key_v1 - v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create api_key table v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_api_key_org_id - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_api_key_key - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_api_key_org_id_name - v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: copy api_key v1 to v2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: Drop old table api_key_v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create dashboard_snapshot table v4
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: drop table dashboard_snapshot_v4 #1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create dashboard_snapshot table v5 #2
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_dashboard_snapshot_key - v5
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_dashboard_snapshot_delete_key - v5
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index IDX_dashboard_snapshot_user_id - v5
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create quota table v1
grafana_1 | 2016/10/24 14:27:06 [I] Migrator: exec migration id: create index UQE_quota_org_id_user_id_target - v1
grafana_1 | 2016/10/24 14:27:06 [I] Created default admin user: admin
grafana_1 | 2016/10/24 14:27:06 [I] Listen: http://0.0.0.0:3000
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
influxdb_1 | => Waiting for confirmation of InfluxDB service startup ...
cadvisor_1 | E1024 14:28:05.841455 1 memory.go:91] failed to write stats to influxDb - Post http://influxsrv:8086/write?consistency=&db=cadvisor&precision=&rp=: dial tcp 172.17.0.3:8086: getsockopt: connection refused
Hi @vegasbrianc
Thanks a lot for the awesome tutorial on monitoring https://www.brianchristner.io/how-to-setup-docker-monitoring/... I'm trying to get the InfluxDB 0.9.x support to work and I cannot even get the data out to InfluxDB...
Could you please verify google/cadvisor#1040 (comment)? Would there be anything you could help here? I'm aware that InfluxDB 0.9.x changed and broke compatibility, so the cAdvisor team accepted this PR and I still can't get to verify it...
thanks a lot
Marcello
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.