huge dips in data being sent to graphite about mysql-statsd HOT 7 CLOSED

Hi John,

Thanks for using mysql_statsd!
Are you sending the data as a gauge, raw or as a delta?
The com_select is an ever increasing number in MySQL so sending it as raw will work in most cases, but to calculate it back to a rate will require you to apply a non-negative derivative. However if you seek for this metric as an ever increasing number it is adviced to be sent as a gauge. Gauge will keep the last stored value until it receives a different value, while a normal metric takes the average of all sent values.

If the sending is exactly the same rate (once per every second) as your statsd daemon sends it (once per 10 seconds by default) to your graphite cluster it works out in most of the cases. However sometimes the timing will be a few milliseconds off and you will miss a beat. Then the statsd daemon will average 9 values over 10 seconds the rate drops.

This is why we introduced the delta value where it stores the last known value in memory and subtracts that from the previously known value and only sends the delta to the statsd daemon. You might still miss a beat in sending metrics, but the average drop is less steep as with the raw value.

I see our example configuration wasn't updated in the latest release, so I'll fix that today. At least this is the release:
mysql-statsd-0.1.2...master
You can change the raw values to delta by changing the r to d.

Does this fix your issue?

from mysql-statsd.

Thank you for the thorough explanation. I changed the values to delta and it seemed to immediately fixed the issue. I've had it running for about half a day and it seems to be working just as expected. I'll comment back here if there's any further issues.

On a side note, do you want some help with the documentation? This likely has to do with my limited knowledge of nodejs, statsd and python, but I has to do some digging to get it to work. I've install mysql-statsd on a handful of Ubuntu 14.04 machines today so would be fairly familiar with all the steps needed to get it up and running.

I would also mention how the configuration file works. I was not totally aware that I could change the metics in the config file - like adding new ones. Again - this is likely a noob issue, but thought it might helpful to mention it how it can be adjusted.

I would love to help with this great project so please let me know what I can do.

Thanks again!

from mysql-statsd.

Good to hear your issue has been resolved now!
I have extended the documentation with a config file section, so I hope everything is clear after my colleagues have merged the pull request. But you are more than welcome to help our with the documentation. 👍

from mysql-statsd.

Still not sure I'm doing this right or using the write date type. I changed the "com_"s to deltas, but noticed some decimal numbers where being sent.

For example the graph below is showing some test queries I ran on the machine. I would expect these numbers to be whole numbers. But is this due to the way statsd is averaging them?

I think switching to the gauge type will fix this issue, but I wanted to check with you first since your update documentation specifically said these com_ values should be delta types.

from mysql-statsd.

Sorry for the late reply. Yes StatsD will average out the number of queries as it is a rate (so queries per second). That means you can get fractions in the end result.
On the flip side if you stick to a gauge you will only see an ever increasing graph, or if you apply the integral function of Graphite a similar rate graph like this one.

from mysql-statsd.

I think we can close this issue now
(will close it next week if no one else replies)

from mysql-statsd.

Yes, thanks for your great support guys!

from mysql-statsd.