Skewed metrics about collectd-carbon HOT 5 CLOSED

My first reaction to your observation is it might be a "problem" with collectd rather than the collectd-carbon. Have you tried using Collectd's built-in RRD writer and seeing if it exhibits the same issue?

Also, CPU values may not be percentages, but rather the number of ticks spent in a state during a measurement cycle. To get the 100% value, you have to add the values of all these tick counts together. If you do that (create a stacked graph in Graphite), the height of the graph should be constant over time. Do you see that?

from collectd-carbon.

I'll put collectd-web on the VM and see what it spits out.
I've compared it with another sender that sends cpu data to graphite and both graphs follow the same trend (they display the same graph shape ). Only that the collectd plugin maximizes at 3000 while the other plugin at 100.
What the maximum that is goes to for you ?

from collectd-carbon.

My personal Ubuntu VM seems to be normalizing the CPU values to around 1000, when summed. However, some sample sets sum to more than 1000, so it doesn't appear exact. I'd have to look at the Collectd CPU code (https://github.com/octo/collectd/blob/master/src/cpu.c) to see what's going on.

FWIW, you can see the raw values dispatched to carbon by packet sniffing. e.g.

$ ngrep -d any port 2003 | grep cpu

Interestingly, the RRD files written by Collectd seem to record values normalized to around 100. It is interesting that the RRD value times the polling interval (10s by default) is what is being sent to Whisper.

I see from my Collectd types.db file that "cpu" is a counter type:

cpu                     value:COUNTER:0:4294967295

The collectd-plugin DeriveCounters config option should come into play here. Let me poke around some more.

from collectd-carbon.

I'm guessing you are seeing values around 3000 because your Collectd polling interval is 30s. Read on for more.

Here is an example of my raw CPU values from Collectd, reported to Python:

Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.user.value = 2018
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.nice.value = 2
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.system.value = 1106
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.idle.value = 100783
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.wait.value = 1825
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.interrupt.value = 0
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.softirq.value = 31
Oct 18 22:39:30 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.steal.value = 0
...
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.nice.value = 2
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.system.value = 1107
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.idle.value = 101781
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.wait.value = 1825
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.interrupt.value = 0
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.softirq.value = 31
Oct 18 22:39:40 ubuntu collectd[1773]: ubuntu-vm-01.cpu.0.cpu.steal.value = 0

The system wasn't doing much when this was measured, so idle time is the only metric that significantly moves. You can see it increased by 998 in 10s.

collectd-carbon with DeriveCounters on simply takes this difference and dispatches that to carbon. This is expected behavior.

RRD does things a little differently. I believe it stores all COUNTER data as derived values internally. Furthermore, it stores the time derivative between 2 values, not the raw difference. This is why you are seeing everything normalized around 100 in RRD. collectd-carbon, however, stores the difference between 2 sample points (with DeriveCounters on), not the time derivative. "DeriveCounters" is somewhat of a misnomer, as it is really merely taking the difference between 2 samples without regards for time.

You can make an argument either way on which is better. The bad news is collectd-carbon doesn't support performing the time derivative. However, it is trivial enough to compute this in Graphite using a scale function (scale by the inverse of the Collectd collection interval).

Because this is expected behavior in the current version, I'm going to close this issue. However, I do feel that dispatching the actual time-derived value would be a useful feature. I'll be opening a new issue for this shortly (I like to keep feature requests separate from possible bug reports).

Does all this make sense?

from collectd-carbon.

As GitHub nicely reports, I've opened issue #9 to track the new feature. I've also updated README.md to further explain the current behavior.

from collectd-carbon.