bulkindexing for probe_events about metrictank HOT 5 CLOSED

NSQ performance will degrade if we delay ACK'ing messages

@woodsaj where do you get this from?
as a reminder for posterity: the problem that drove #11 was that raintank-metric was losing metrics due to crashes and we couldn't simply delay acks to rabbit until after safe delivery. the primary reason we switched from rabbitmq to nsq is that nsq handles late ACK's well, so that we could use its diskqueue for metrics messages while we try to store batches of metrics into kairos, instead of being incentivized to prematurely (unsafely) ack in raintank-metric.

the story here, although we have more individual, smaller messages, is basically the same though.
just to be sure i asked again on the nsq-users email list:
https://groups.google.com/forum/#!topic/nsq-users/8TpXCYGcEFI
and they confirm there should be no issues.

from metrictank.

I am certainly in no position to argue with the NSQ developers.

my assumption was based on the fact that nsq has a bounded memory footprint. Delaying acks will cause (depending on memory consumption) messages to be moved out of RAM, and i would assume acking them would require loading them back into RAM. This could lead to high IO as messages are read from the diskqueue, as the read pattern would be random rather then sequential.

from metrictank.

looking at the code, we've implemented bulkindexing for probe events, but it acks when it succeeds to add to the bulkindexing queue. so if nsq_probe_events_to_es crashes or restarted, we may see dataloss. especially if ES is down, even if temporarily.

from metrictank.

@Dieterbe for events, nsq messages are only acked after we get confirmation from ES that the points have been written.

What makes you think otherwise?

from metrictank.

looks like i was wrong. i didn't see the inprogresswritequeue stuff. i think we can close this

from metrictank.