"Too many files" - TCP connection exhaustion about graphite_exporter HOT 4 CLOSED

Considering the entire purpose of the graphite exporter is to accept connections from other servers, a limit here would be solving the wrong problem. I'd suggest bumping the ulimit, and fixing those servers.

from graphite_exporter.

Actually it's starting to look like this might be on the exporter end - we're sending our graphite metrics via Diamond, which has its own implementation of the protocol, and it sure looks like the graphite exporter is holding open connections which don't even have a process on the other end anymore.

I'll do some more digging and try to really isolate the issue.

from graphite_exporter.

Hmm, can you share a goroutine dump?

from graphite_exporter.

Well, this is the strangest thing. We put a carbon relay in between our Diamond collectors and the graphite_exporter, and as an apparent side effect of that, the number of sockets on the graphite_exporter has been totally stable. (But we also haven't seen a leak on the relay, so it may not be as simple as an issue with the Diamond side of the connections, either.)

In any case, we aren't affected by this issue any longer... if anybody else has the same issue, throwing a relay in front is a pretty easy workaround. (I used https://github.com/graphite-ng/carbon-relay-ng)

from graphite_exporter.