Bridge Crashes When Counter Decreases about statsd_exporter HOT 8 CLOSED

It crashes in this line:

https://github.com/prometheus/statsd_bridge/blob/master/bridge.go#L217

That probably means that NSQ sends counter increments with negative values (which Prometheus forbids). Can you confirm? Does NSQ internally "reset" its counters by decrementing them by their current value?

I agree the statsd-bridge shouldn't crash in that case, but probably should just ignore negative counter increments.

from statsd_exporter.

I guess you could say they decrement by their current value, but really they just all reset to 0 upon a restart.

from statsd_exporter.

The key thing is, they seem to send negative counter increments via the StatsD protocol. I'm not sure how much sense that makes. Usually you don't want a counter reset to be reflected as a negative rate (which I think will happen statsd-wise in this case). You'd just want to ignore it in the rate calculation. In any case, I'm all for simply ignoring negative counter increments in the bridge (while logging a warning about it). Would you be up for sending a PR?

from statsd_exporter.

This is still a problem

from statsd_exporter.

@juliusv if nsq restarts its stats reset, it isn't sending negative values, but it does send values that could be less than what it had originally.

from statsd_exporter.

This one bit me today as well (we've got a proof of concept prometheus server up, and I just repointed one of our application servers to the statsd_exporter). I don't know golang, but my colleague does, so we'll see if he can patch and issue a PR.

from statsd_exporter.

That would be awesome.

from statsd_exporter.

Should this be closed given that #33 has been merged ?

from statsd_exporter.