Since 0.4, the config file format has changed to YAML as well as several other features. I'm currently running a version close to master in production to ensure I can use histograms.

Is it time to do a 0.5 release? What other things would we liked to be merged before then? I'm happy to do the mechanics of a releases if we decide its time.

Any plans for a new release?

ERRO[0785] Error building event on line docker-lb2.fabio--garbage.tacostand_com./.12_117_14_4_21506.mean:79292673.35|gf: Bad stat type gf source=exporter.go:51

Is an example of a badly formed statsd metric for the non-existent type gf which raises an error level log message from statsd_exporter. This doesn't smell like an error and should probably be downgraded to info level.

https://dave.cheney.net/2015/11/05/lets-talk-about-logging

Context

Our statsd server collects a large number of time series for many different hosts. Most of these metrics are in the form of:

stats.<namespace>.<host-id>.<metric_namespace>.<metric_name>
or
stats.<namespace>.<host-id>.<metric_namespace>.<metric_namespace_2>.<metric_namespace...>.<metric_name>

These metrics when collected should simply be:
<escaped_metric_name>{host_id=<host_id>,namespace=<namespace>_<metric_namespace>}
and
<escaped_metric_name>{host_id=<host_id>,namespace=<namespace>_<metric_namespace>_<metric_namespace2>_<metric_namespace...>}

This is hard because our current mappings won't allow for dynamic names and also don't allow for full regex on metrics.

Solution

I suggest we write the mapping config with full golang regex. This allows full control over how the metric's should be mapped.

Example Config

# This creates the most basic mapping. It escapes the metric and then exports it 
(.*)
name=$1

# This creates the more advanced mapping needed above. All the groups would be escaped automatically. 
stats.([^.]+)\.([^.]+)\.(.+)\.([^.]+)$
name=$4
namespace=$1_$3
host=$2

Hopefully we can kick off the discussion

Once statsd_bridge recieves a metric, it seems to hold a bucket for it (until the process dies).
Sometimes a metric is getting sent once in a while and does not require the persistency, is that by design that it will never clean/remove buckets with Nan values or with Nan values for a long period?

Problem
It appears that the initFromFile for the mapping config file is only read and then passed to initFromString. That method does not appear to have any way to handle comments in the string (or in this specific thought, the mapping file)

Possible Solution
Support golang style (//) comments in the initFromString and ignore them in the case statement in the if statement

I have a statsd producer that's emitting a constant counter type metric

i.e:

myhost.myexec.http.status.404.count:2329|c
myhost.myexec.http.status.200.count:30|c
myhost.myexec.http.status.304.count:10|c

Because the metric is a counter in the origin system, if I don't get any requests during the scraping period, it'll send again the same value the next time.

I've added the following mapping in my statsd_exporter to use prometheus dimentions like this:

mappings:
- match: ^([^.]*)\.([^.]*)--http.status.([^.]*).count
  match_type: "regex"
  name: "http_status_count"
  labels:
    code: "$3"

So the ending metric is something like:

http_status_count{code="200"} 1557
http_status_count{code="304"} 18
http_status_count{code="404"} 6987

What's happening is that statsd_exporter is constantly adding the values that the origin is pushing even though they don't change. So what I end up seing in the prometheus metrics is a constant increment even though the metrics don't change.

Any ideas?

If there is no new-line after mappings, the last mapping is invalid. For example, this configuration file:

logstash.*.*.*.events
name="logstash_events_total"
source="$1"
instance="$2"
stage="$3"<no newline here>

... will not work. Adding the newline, it starts to work.

To be consistent with the rest of our exporters, this repository should be renamed to the statsd_exporter as it takes in data from a non-prometheus system and exports it in a prometheus format.

By contrast a "bridge" would take data in some prometheus format, and output it in a non-prometheus format.

When I tried taking parts of the parsed output and use it for the metric name, for example: "name": "$2_$3", it seems that statsd_bridge fail to start if I configure it and complains about it.
Using a static name it works.

512 bytes are ok for the wild Internet. For local network we can send udp packets up to 1432 bytes or greater. For example it's very important if we use buffered client (github.com/cactus/go-statsd-client/statsd/client_buffered.go). So I suggest increasing buf size to 2-8k.
Also MTU is important for client. Udp packet can be split or dropped by ISP because of size. Server shouldn't evaluate MTU.

{code}
func (l *StatsDListener) Listen(e chan<- Events) {
// TODO: evaluate proper size according to MTU
var buf [512]byte
for {
n, _, err := l.conn.ReadFromUDP(buf[0:])
if err != nil {
log.Fatal(err)
}
l.handlePacket(buf[0:n], e)
}
}
{code}

I'm not sure that I have good proposal to solve this, but I /just/ ran into the issue when mapping some statsd metrics that I cannot define a metric http_requests_total because it is already defined by default.

I suppose if an existing metric is redefined by the mapping that the user defined mapping should win out. I'm less sure what to do with the internal telemetry/instrumentation which had originally defined the metric.

When reading from TCP, we abort and close the connection if any line does not fit into the bufio.Reader buffer. This is really not necessary, and the standard library has higher-level methods to reliably read lines from a stream.

Any solution should make sure to process the last line in the stream, even if it does not end in a newline.

Hi,
the recommended usage of statsd_exporter from README.md is with a statsd server relaying metrics to it.
Our current statsd deployment involves a central statsd traffic receiving traffic for "global aggregation" purposes, though we'd like to gradually move to Prometheus. During the transition we'd co-locate statsd_exporter on the same host running the service and scrape from Prometheus while still keeping the statsd traffic to the central server.

Would you be interested in a PR to add relaying capabilities to statsd_exporter itself? I think this would simplying deployment in cases where the exporter acts "inline" with statsd traffic, without an additional statsd repeater.

Thanks!

Gauges should provide functionality to increase/decrease the metric by the specified value, not just always set the metric to the value.

For instance;

myGauge:1|g should set this gauge to 1
myGauge:+1|g should increment the gauge by 1
myGauge:-1|g should decrement the gauge by 1

Maybe after #13

@discordianfish

Currently, the statsd_bridge uses summary as the mapping for statsd timers and I am wondering what you think about allowing the bridge to decide between using summary or histogram for statsd timers?

Some backstory:
As our clients are already instrumented with statsd, we're considering using the statsd_bridge as an initial step to migrate our webservices metrics into prometheus and validate our setup.

Long term, we plan to use the python-client (and contribute to it) but in the very short term I am somewhat blocked on how to deal with pre-fork servers like gunicorn in a clean way. The statsd_bridge would provide us with an IPC mechanism to send metrics from multiple instance/processes and expose it to prometheus. With the histogram support, that would allow us to run statsd_bridge per instance and expose a per instance metrics endpoint (a very desirable feature).

If that's a direction that is acceptable, a few things will need to be designed/implemented:

• Are "summary vs histogram" a runtime configuration or something that could be set on a per mapping basis?
• If the choice can be made on per mapping basis, should that be extended to handle bucket sizes or use a default set of buckets?

To get more familiar with the code, I made a small test branch to validate the histogram implementation:
https://github.com/prometheus/statsd_bridge/compare/master...marcusmartins:histogram_support?expand=1

Readme says:

We recommend this only as an intermediate solution and recommend switching to native Prometheus instrumentation in the long term.

I would like this somewhat clarified. For forking web servers such as Passenger, Unicorn and Puma in the Ruby ecosystem, requests are round robined between process forks. This means you have no simple way of sharing state between forks and thus have a hard time collating data for an exporter.

In this kind of setup a solution like statsd_exporter seems reasonable as the alternatives are all similarly messy. Be it sharing a pipe between all forks or nmaping a file and so on.

I think the Readme should clarify that this is a completely legitimate solution for specific use cases such as aggregation of data from forked processes that all share an HTTP listener.

Please add godep manifest and make new release.
If checkout version 0.1.0 and try make godep get i get this result

github.com/howeyc/fsnotify (download)
github.com/prometheus/client_golang (download)
package github.com/prometheus/client_golang/model: cannot find package "github.com/prometheus/client_golang/model" in any of:
    /usr/local/go/src/github.com/prometheus/client_golang/model (from $GOROOT)
    /go/src/github.com/prometheus/client_golang/model (from $GOPATH)
github.com/beorn7/perks (download)
github.com/golang/protobuf (download)
github.com/prometheus/client_model (download)
github.com/prometheus/common (download)
github.com/matttproud/golang_protobuf_extensions (download)
github.com/prometheus/procfs (download)
godep: exit status 1

Hi,
I'm seeing a ratio of 10k metrics per 100M of RAM, for production metrics. Interested to hear first if that's normal, and then, what's a good strategy for handling 1M metrics (this also affects the metrics endpoint polled by prom, obviously).

In my tests for 1M metrics, I've gotten 28GB of RAM and growing, and all cores max out (had to kill the process).

README contains:

Metrics that don't match any mapping in the configuration file are translated into Prometheus metrics without any labels and with certain characters escaped (_ -> __; - -> __; . -> _)

But in reality metrics with names metric_name1, metric-name2, metric.name3 become metric_name1, metric_name2, metric_name3 accordingly.

there is the code to reproduce this doc vs reality mismatch https://gist.github.com/nordicdyno/061a50dc29336dc6a60d4d0470edd151

We had some developers on a new project starting to integrate statsd support in their application. statsd_exporter 0.3.0 is crashing constantly with this error message:

FATA[0000] A change of configuration created inconsistent metrics for "query_timer". You have to restart the statsd_exporter, and you should consider the effects on your monitoring setup. Error: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "query_timer", help: "Metric autogenerated by statsd_exporter.", constLabels: {after_date="2017-01-19 00:00:00-05:00",application="…",component="…",environment="testing",server="…",sub_component="…"}, variableLabels: []} has different label names or a different help string  source=exporter.go:137

If I'm reading that correctly, this due to them having two versions of the app sending inconsistent metric values but while an error in this situation seems reasonable it actually causes the statsd_exporter process to crash.

#66 was recently merged and I'm a huge fan of the new yaml mappings, however I think it is probably a bad idea to have the name label have magic behavior in setting the metric name in the exporter. I /think/ it probably makes more sense to have a metric_name parameter one level higher in the mapper config. This should disambiguate the result of the mapping AND allow for, if desired, a mapping which sets a label called name.

Proposed change...
this:

mappings:
- match: test.case.*
  labels:
    name: "test_case_total"
    blargle: "$1"

Would become this:

mappings:
- match: test.case.*
  metric_name: "test_case_total"
  labels:
    blargle: "$1"

statsd_exporter_packets_total does not count packets. It counts events, and the dimensions are not used consistantly. dogstatsd metrics get double counted as both legal and dogstatsd.

networkStats.WithLabelValues("legal").Inc() should be moved out of the per-line loop.
I can PR this, but the other counters need some thought too, the various forms of invalid should probably be on events (lines), rather than packets.

Hello!
There is situation when statsd_exporter won't start if there is no newline in config file.
We are using kubernetes 1.6 and unfortunately it creates config files from configmaps
with no newline at the end of file... Is it possible to eliminate this restriction?

client_golang at https://github.com/prometheus/statsd_exporter/tree/master/vendor/github.com/prometheus/client_golang is not updated with the latest files from https://github.com/prometheus/client_golang
This causes discrepancies in the metrics generated by statsd and prometheus.
The pushgateway floods it's stdout with messages like below
time="2017-08-30T08:05:00Z" level=warning msg="Metric families 'name:"go_memstats_sys_bytes" help:"Number of bytes obtained by system. Sum of all system allocations." type:GAUGE metric:<label:<name:"instance" value:"core-services-ad-level-canary-1654703993-21nr1" > label:<name:"job" value:"target_9102" > gauge:<value:1.9605752e+07 > timestamp_ms:1504080298125 > ' and 'name:"go_memstats_sys_bytes" help:"Number of bytes obtained from system." type:GAUGE metric:<label:<name:"instance" value:"core-services-host-level-ds-blfqn" > label:<name:"job" value:"target_9126" > gauge:<value:1.44943352e+08 > timestamp_ms:1504080291333 > metric:<label:<name:"instance" value:"core-services-host-level-ds-gxkg4" > label:<name:"job" value:"target_9126" > gauge:<value:1.83412984e+08 > timestamp_ms:1504080296389 > ' are inconsistent, help and type of the latter will have priority. This is bad. Fix your pushed metrics!" source="diskmetricstore.go:114"

from time to time, looks like the metrics endpoint dies and start returning 500 errors, and only way to restore it is to restart the pod it's on.
Iogs are empty and I didn't see any options to switch on a more verbose logging.
what can I do in order to provide a more detailed log and help debug it?

I'd like to prefix my metrics to add a namespace. For example, I send a metric to statsd as "login". It'd be nice to have prometheus read that as appname_login. If I have statsd prefix the metric myself as appname_ then the mappings break. Without the prefix, the following works:

login.*
name="login"
type="$1"

After the prefix

appname_login.*
name="login"
type="$1"

Doesn't seem to match. And I see appname_login_attempt 1.

Ref. prometheus/common#38

The current behavior of providing a help text like
# HELP test_case_total Metric autogenerated by statsd_exporter.

falls pretty far short of the intent of informing the consumer of the metric what the metric represents in easy language. This should be a very small modification that would improve usability.

I'm sending statsd metrics from Openstack Swift to the statsd_exporter and seeing error messages like:

time="2017-01-06T10:18:08Z" level=error msg="Illegal sampling factor for non-counter metric on line swift.object-server.REPLICATE.timing:0.654935836792|ms|@0.1" source="exporter.go:390"

The statsd_exporter only allows sampling factors for counter metrics, but it should also do so for timer metrics. See for example https://github.com/etsy/statsd/blob/master/docs/metric_types.md#timing for upstream documentation that references sampling factors for timer metrics.

I get the following metrics wihtout mapping

# HELP consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal Metric autogenerated by statsd_exporter.
# TYPE consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal summary
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal{quantile="0.5"} 1.453363
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal{quantile="0.9"} 1.556893
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal{quantile="0.99"} 1.556893
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal_sum 4.627288
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal_count 3

I want the metric name like so

consul_consul_dns_domain_query

I have a statsd_mmaping.conf like so

consul.consul.dns.domain.query.*
name="consul_dns_domain_query"

But still the metrics show

consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal{quantile="0.5"} 1.453363
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal{quantile="0.9"} 1.556893
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal{quantile="0.99"} 1.556893
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal_sum 4.627288
consul_consul_dns_domain_query_ip_10_113_143_167_ec2_internal_count 3

While trying to use this exporter with some particularly heinous 3rd party software I quickly ran into the limitation that I would like to be able to define mappings which are more precisely defined than what is allowed in the current globbing match syntax. Looking through the code, the globbing is actually handled internally by being translated to regex.

I'd propose that there should be a configuration option available utilizing the newer yaml mapper configuration that allows for selection between match_type with a value of either glob or regex.

I've been testing the changes necessary to implement this and will have a PR shortly.

CircleCI 1.0 will be shut down on August 31. Resources

As reported in #110, we don't expose the logging flags from github.com/prometheus/common/log. That has switched to kingpin, like the other Prometheus projects. We should switch over too, so that we can use log.AddFlags.

In the README it is noted that the -log.format command line parameter is supported to log in the JSON format.

When I try to run with -log.format=logger:stdout?json=true I get the error flag provided but not defined: -log.format

When checking the -help, -log.format is not listed at all...

docker run -i --rm prom/statsd-exporter:v0.5.0 -help                                                                                                                                            
Unable to find image 'prom/statsd-exporter:v0.5.0' locally
v0.5.0: Pulling from prom/statsd-exporter
aab39f0bc16d: Pull complete 
a3ed95caeb02: Pull complete 
2cd9e239cea6: Pull complete 
628ef644fd9a: Pull complete 
Digest: sha256:d08dd0db8eaaf716089d6914ed0236a794d140f4a0fe1fd165cda3e673d1ed4c
Status: Downloaded newer image for prom/statsd-exporter:v0.5.0
Usage of /bin/statsd_exporter:
  -statsd.listen-address string
    	The UDP address on which to receive statsd metric lines. DEPRECATED, use statsd.listen-udp instead.
  -statsd.listen-tcp string
    	The TCP address on which to receive statsd metric lines. "" disables it. (default ":9125")
  -statsd.listen-udp string
    	The UDP address on which to receive statsd metric lines. "" disables it. (default ":9125")
  -statsd.mapping-config string
    	Metric mapping configuration file name.
  -statsd.read-buffer int
    	Size (in bytes) of the operating system's transmit read buffer associated with the UDP connection. Please make sure the kernel parameters net.core.rmem_max is set to a value greater than the value specified.
  -version
    	Print version information.
  -web.listen-address string
    	The address on which to expose the web interface and generated Prometheus metrics. (default ":9102")
  -web.telemetry-path string
    	Path under which to expose metrics. (default "/metrics")

The default tag for Docker images is latest but a new tag went up under master on DockerHub. This means the pull command fails:

$ docker pull prom/statsd-exporter
Using default tag: latest
Pulling repository docker.io/prom/statsd-exporter
Tag latest not found in repository docker.io/prom/statsd-exporter

One must now run docker pull prom/statsd-exporter:master instead.

It would be lovely to have it published under the latest tag again as it would otherwise mean a lot of config file updates. Thanks!

This exporter seem to be loosing metrics on restarts, can we get persistence for them as it's done in the push gateway?

The default behavior of passing unmapped metrics through can result in really polluted metric namespaces when using software that generates a large number of dynamic metrics OR a large number of metrics that are irrelevent to the user's interest. I'd like to be able to switch the behavior in the config to blackholing unmapped metrics.

Either my day is far too long today or PR #66 has a little mathematic error which I simply have not seen in my test environment because my setup was too little and fast.

If I push some statsd timers to statsd and let them export with statsd_exporter all metrics are send to the +Inf bucket. statsd uses metrics in milliseconds per default but the histogram buckets are set to seconds.

This came to my attention when I deployed the current master branch of statsd_exporter to one of our production host. To reproduce it, simply send some "realistic" metrics to statsd, like:

echo "test.timing:100|ms|@0.1" | nc -u -w0 127.0.0.1 8125
echo "test.timing:320|ms|@0.1" | nc -u -w0 127.0.0.1 8125
echo "test.timing:500|ms|@0.1" | nc -u -w0 127.0.0.1 8125
echo "test.timing:900|ms|@0.1" | nc -u -w0 127.0.0.1 8125

It will export to:

test_timing_bucket{job="debug.goethe.local",le="0.005"} 0
test_timing_bucket{job="debug.goethe.local",le="0.01"} 0
test_timing_bucket{job="debug.goethe.local",le="0.025"} 0
test_timing_bucket{job="debug.goethe.local",le="0.05"} 0
test_timing_bucket{job="debug.goethe.local",le="0.1"} 0
test_timing_bucket{job="debug.goethe.local",le="0.25"} 0
test_timing_bucket{job="debug.goethe.local",le="0.5"} 0
test_timing_bucket{job="debug.goethe.local",le="1"} 0
test_timing_bucket{job="debug.goethe.local",le="2.5"} 0
test_timing_bucket{job="debug.goethe.local",le="5"} 0
test_timing_bucket{job="debug.goethe.local",le="10"} 0
test_timing_bucket{job="debug.goethe.local",le="+Inf"} 4

Can someone confirm this or am I doing something wrong?

This is probably related to #13 .. but the docs currently reference a docker image at prom/statsd-exporter . This project exists on docker but there are no images uploaded. Could we get an image up there?

Basically I'm forwarding nsq stats to statsd, with statsd repeating to the bridge. When NSQ restarts its stats restart, or when a NSQ reader disconnects and reconnects the stats reset, therefore the counter would reset.

I was told in IRC that Prometheus itself understands concepts like this.

Is there a way we can make the bridge not crash in this scenario?

2015/04/10 23:17:12 Starting StatsD -> Prometheus Bridge...
2015/04/10 23:17:12 Accepting StatsD Traffic on :9125
2015/04/10 23:17:12 Accepting Prometheus Requests on :9102
panic: counter cannot decrease in value

goroutine 16 [running]:
runtime.panic(0x6fef00, 0xc20870fb10)
        /usr/src/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/prometheus/client_golang/prometheus.(*counter).Add(0xc208196200, 0xc0ef876000000000)
        /go/src/github.com/prometheus/client_golang/prometheus/counter.go:69 +0xf9
main.(*Bridge).Listen(0xc20800f500, 0xc2080a2000)
        /go/src/app/bridge.go:217 +0x617
main.main()
        /go/src/app/main.go:127 +0x5e3

goroutine 19 [finalizer wait, 8 minutes]:
runtime.park(0x41a430, 0x952a78, 0x950cc9)
        /usr/src/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x952a78, 0x950cc9)
        /usr/src/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
        /usr/src/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
        /usr/src/go/src/pkg/runtime/proc.c:1445

goroutine 26 [IO wait, 5 minutes]:
net.runtime_pollWait(0x7f1489fffb20, 0x72, 0x0)
        /usr/src/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080ba140, 0x72, 0x0, 0x0)
        /usr/src/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080ba140, 0x0, 0x0)
        /usr/src/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc2080ba0e0, 0x823cd8, 0x0, 0x7f1489ffe520, 0xb)
        /usr/src/go/src/pkg/net/fd_unix.go:419 +0x343
net.(*TCPListener).AcceptTCP(0xc20803a0b8, 0x4dae33, 0x0, 0x0)
        /usr/src/go/src/pkg/net/tcpsock_posix.go:234 +0x5d
net/http.tcpKeepAliveListener.Accept(0xc20803a0b8, 0x0, 0x0, 0x0, 0x0)
        /usr/src/go/src/pkg/net/http/server.go:1947 +0x4b
net/http.(*Server).Serve(0xc2080044e0, 0x7f1489ffffb0, 0xc20803a0b8, 0x0, 0x0)
        /usr/src/go/src/pkg/net/http/server.go:1698 +0x91
net/http.(*Server).ListenAndServe(0xc2080044e0, 0x0, 0x0)
        /usr/src/go/src/pkg/net/http/server.go:1688 +0x14d
net/http.ListenAndServe(0x776410, 0x5, 0x0, 0x0, 0x0, 0x0)
        /usr/src/go/src/pkg/net/http/server.go:1778 +0x79
main.serveHTTP()
        /go/src/app/main.go:36 +0x82
created by main.main
        /go/src/app/main.go:105 +0x2b8

goroutine 27 [IO wait]:
net.runtime_pollWait(0x7f1489fffbd0, 0x72, 0x0)
        /usr/src/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc20802bf00, 0x72, 0x0, 0x0)
        /usr/src/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc20802bf00, 0x0, 0x0)
        /usr/src/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).readFrom(0xc20802bea0, 0x7f1489e5fd38, 0x200, 0x200, 0x0, 0x0, 0x0, 0x7f1489ffe520, 0xb)
        /usr/src/go/src/pkg/net/fd_unix.go:269 +0x3db
net.(*UDPConn).ReadFromUDP(0xc20803a0a8, 0x7f1489e5fd38, 0x200, 0x200, 0xc2080a2000, 0x0, 0x0, 0x0)
        /usr/src/go/src/pkg/net/udpsock_posix.go:67 +0x129
main.(*StatsDListener).Listen(0xc20803a0b0, 0xc2080a2000)
        /go/src/app/bridge.go:288 +0x8b
created by main.main
        /go/src/app/main.go:116 +0x422

goroutine 28 [select, 8 minutes]:
main.watchConfig(0x7fff43d6af43, 0xd, 0xc20800f200)
        /go/src/app/main.go:77 +0x766
created by main.main
        /go/src/app/main.go:124 +0x5bb

goroutine 30 [syscall, 8 minutes]:
syscall.Syscall(0x0, 0x5, 0xc208125e98, 0x10000, 0x0, 0x0, 0x0)
        /usr/src/go/src/pkg/syscall/asm_linux_amd64.s:21 +0x5
syscall.read(0x5, 0xc208125e98, 0x10000, 0x10000, 0x0, 0x0, 0x0)
        /usr/src/go/src/pkg/syscall/zsyscall_linux_amd64.go:838 +0x75
syscall.Read(0x5, 0xc208125e98, 0x10000, 0x10000, 0x0, 0x0, 0x0)
        /usr/src/go/src/pkg/syscall/syscall_unix.go:136 +0x5c
github.com/howeyc/fsnotify.(*Watcher).readEvents(0xc2080041e0)
        /go/src/github.com/howeyc/fsnotify/fsnotify_linux.go:219 +0x133
created by github.com/howeyc/fsnotify.NewWatcher
        /go/src/github.com/howeyc/fsnotify/fsnotify_linux.go:126 +0x288

goroutine 31 [chan receive, 8 minutes]:
github.com/howeyc/fsnotify.(*Watcher).purgeEvents(0xc2080041e0)
        /go/src/github.com/howeyc/fsnotify/fsnotify.go:21 +0x51
created by github.com/howeyc/fsnotify.NewWatcher
        /go/src/github.com/howeyc/fsnotify/fsnotify_linux.go:127 +0x2a0

Hi @matthiasr,
I still have a follow-up question. I plan to use short living batch jobs, long living batch jobs (they will be converted to streaming jobs over time) and spark streaming jobs. Can't I not just simply push all metrics to the pushgateway rather than using statsd_exporter at all and prometheus scrapes the metrics from there? What speaks against it?
For example: Imagine statsd_exporter restarts, all metrics get lost?
Thanks,
Arnold

hi,

do you currently support metric type Sets?

StatsD supports counting unique occurences of events between flushes, using a Set to store all occuring events.

uniques:765|s
If the count at flush is 0 then you can opt to send no metric at all for this set, by setting config.deleteSets.

spec:
https://github.com/etsy/statsd/blob/master/docs/metric_types.md

statsd will automatically reset counters when publishing to graphite and i'm looking for a similar feature in the exporter too. Did i miss the settings for this or isn't this available?

Currently we use counters which get reset on every publish, so we can use that values to detect inactivity or outages (when the metric is no longer updated). Technically it's a gauge with automatic expiry when not updated for a while.

Is there an option to expire or reset after a certain amount of time? Otherwise we'll need to look into the full prometheus instrumentation migration which was actually planned for later this year.

Sometimes (#122) a data source sends metrics that are not quite correct – for example, sending "c" statsd events but actually sending an all-time count every time. It would help to correct for this if the mapping would allow to override the automatically determined metric type. This would need another attribute in the mapping configuration, and a clear documentation of the implications.

How can I scale stastd_exporter in order to improve availability?

I was considering to have a load balancer pointing to two stastd_exporter instances. Then I would query for the sum of every instance, given a certain metric. This approach works fine with counters, but fails with gauges.

Would you guys have a suggestion?

The README is outdated in several places, but especially around the new TCP support.

@jwfang

As I understand statsd's repeater mode, it forwards the unprocessed traffic it receives from statsd clients. This makes me think that instead of continuing to run statsd and having it repeat traffic to statsd-bridge, statsd-bridge can replace statsd and receive traffic directly from statsd client libraries. If this is correct, it would be nice to update the Overview section of the README to mention this style of usage.