mojodna / metricsd Goto Github PK

I attempted to add support for this myself (no real Scala experience here) but after getting down into the point where coda's metrics library started dumping stuff to graphite by grouped under the metric type, I decided to open an issue.

We like to group our metrics in graphite under "stats.company.. Right now the top-level of metrics is hardcoded. I was able to change THAT part but as I discovered the actual type of metric is being injected I THINK by coda's stuff in GraphiteReporter.

I'm thinking this would be a bigger change than the approach I was looking at. I'm willing to give it a shot (been looking for an itch to scratch in Scala) but wanted to see your thoughts.

I'd really like to use metricsd as I'm keen on the additional data types.

Hey, mojodna,
Sorry to put this in the issue list. If you have a maillist, please let know. I will move it.
Is it possible to remove the dependency to metrics lib? Based on the reading from here http://readthedocs.org/docs/graphite/en/latest/feeding-carbon.html. It should be very straight forward to write to carbon directly and we can choose different protocol: plaintext, pickle or even AMQP. Can you please explain to me what's the benefits you use metrics lib here?
Thank you.
Yongjun Rong

Hi @mojodna and thanks for your work on metricsd, it's a really useful tool :)

It seems that there's been no activity on this project since Feb 1, 2013 though, so I'm wondering if you still have the time to maintain it?

Best,
Jan

This other project called graphite-relay [https://github.com/markchadwick/graphite-relay#readme] can send timestamp also, can metricsd do this?

Thanks,

Currently every histogram creates the following metrics

75percentile
95percentile
98percentile
999percentile
99percentile
max
mean
median
min
stddev

How can I change this list to say only feature 95percentile, max, mean, median, min ?

Once metricsd has been started, it logs any and all metrics that have been sent to it. Sometimes this is annoying, like when you're testing and generate metrics with the wrong name. Deleting them from the filesystem doesn't work, as metricsd will keep flushing the last values sent.

When I send some very big values, and then send normal values, the percentiles won't change to normal sometimes in a very long time.

See the image. The yellow line is the values I'm sending, and the blue line is 95percentile. And I suppose it should decline after the peak. As histograms are biased to the last 5 min data values. But the 95percentile is a fixed value for a long time. After about 25min, it declines.

In my real usage, when there is a very very very high peak, this kind of impact will be much longer (for many days 95percentile keeps to be a fixed value).

Would it be better if percentiles can be configured to be based on past few minutes, instead of using the biased algorithm from Metrics?

In order for metric deletion to be more useful, metricsd should have a mechanism to list metrics that it knows about.

I'm using metricsd-0.4.1-jar-with-dependencies.jar with default settings under Ubuntu 10.04

counter increment works OK for me.
But when send message as "test:-10|c" will get this:

ERROR [2012-02-24 10:25:57,266] net.mojodna.metricsd.server.MetricsServiceHandler: Exception in MetricsServiceHandler
! scala.MatchError: test:-10|c (of class java.lang.String)
! at net.mojodna.metricsd.server.MetricsServiceHandler$$anonfun$messageReceived$1.apply(MetricsServiceHandler.scala:33)
! at net.mojodna.metricsd.server.MetricsServiceHandler$$anonfun$messageReceived$1.apply(MetricsServiceHandler.scala:31)
! at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
! at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
! at net.mojodna.metricsd.server.MetricsServiceHandler.messageReceived(MetricsServiceHandler.scala:30)
! at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
! at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
! at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
! at org.jboss.netty.channel.socket.nio.NioDatagramWorker.read(NioDatagramWorker.java:413)
! at org.jboss.netty.channel.socket.nio.NioDatagramWorker.processSelectedKeys(NioDatagramWorker.java:342)
! at org.jboss.netty.channel.socket.nio.NioDatagramWorker.run(NioDatagramWorker.java:255)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
! at java.lang.Thread.run(Thread.java:636)

When metricsd starts up, it should load current counter values from carbon (Graphite).

Timer metrics also provide a rate meter. That is:

Timer = histogram + meter

These should be split, which probably means standardizing on a timeunit, probably milliseconds.

MetricsD is unable to parse the following metric
Foo.Web.ServiceHost.QueueWorker.ExternalAnalyticsEventMessage.PendingMessages:3.000000000000000|g

ERROR [2013-11-21 00:36:08,585] net.mojodna.metricsd.server.MetricsServiceHandler: Exception in MetricsServiceHandler
! scala.MatchError: Foo.Web.ServiceHost.QueueWorker.ExternalAnalyticsEventMessage.PendingMessages:3.000000000000000|g (of class java.lang.String)
! at net.mojodna.metricsd.server.MetricsServiceHandler$$anonfun$messageReceived$1.apply(MetricsServiceHandler.scala:34)
! at net.mojodna.metricsd.server.MetricsServiceHandler$$anonfun$messageReceived$1.apply(MetricsServiceHandler.scala:32)
! at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
! at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
! at net.mojodna.metricsd.server.MetricsServiceHandler.messageReceived(MetricsServiceHandler.scala:31)
! at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
! at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
! at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
! at org.jboss.netty.channel.socket.nio.NioDatagramWorker.read(NioDatagramWorker.java:98)
! at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)

Either via syslog or HTTP (http://log-shuttle.io/) using l2met-style measure.* elements (more).

When a gauge hasn't been updated, its value should be reported as 0 rather than its most recent value.