Investigate possible long times between message accepting and writing to ElasticSearch about graylog2-server HOT 5 CLOSED

We've encountered this issue with v0.12.0.
If you search for "missing messages" in the forums you'll find a lot of others that encountered this issue, so I think it should be solved...
From my investigation it seems that the problem is with OutputBufferProcessor which uses its own buffer and write message batches. The last batch is kept in its buffer forever, until the next message arrives. If no more messages arrive, the batch is not written to the proper output.

from graylog2-server.

OutputBufferProcessor.onEvent:
if (endOfBatch || buffer.size() >= server.getConfiguration().getOutputBatchSize()) {
...
}

I've confirmed that setting output_batch_size = 1 resolves this issue (just for testing, since it cannot be done in a high-volume environment due to performance penalty)

from graylog2-server.

I encountered this issue, and can reproduce it with a clean install of the latest release candidate - graylog2-server-0.20.0-rc.2.

I am submitting large batches (several thousand messages each) to the server using a Gelf TCP input.

Here is what I observe:

First, I submit a batch of 7,350 messages via Gelf TCP input.

graylog2-server node shows:

• 0 messages in input buffer

  org.graylog2.buffers.ProcessBuffer.incomingMessages: 7353 events
  org.graylog2.buffers.ProcessBuffer.cachedMessages: 0 events
  org.graylog2.buffers.ProcessBuffer.rejectedMessages: 0 events

• 0 messages in output buffer

  org.graylog2.buffers.OutputBuffer.incomingMessages: 7353 events
  org.graylog2.buffers.OutputBuffer.cachedMessages: 4725 events
  org.graylog2.buffers.OutputBuffer.rejectedMessages: 0 events

However:

The System -> Indices page reports:

“1 indices with a total of 6,902 messages under management. Current write-active index is graylog2_0.”

6,899 of these are are my messages (plus 3 system messages that were not submitted by me).

At this point, 451 messages have not made it into ElasticSearch, and will remain in limbo indefinitely until the next batch of messages is submitted

Next, I submit a second batch of 3,422 messages via the Gelf TCP input.

Now:

• All 7,350 of the messages from the first batch are returned in search results from ElasticSearch
• Only 3,215 / 3,422 messages from the second batch are returned in search results from ElasticSearch

from graylog2-server.

Based on the descriptions here, it seems that implementing Issue #317 may resolve this issue.

from graylog2-server.

This has been fixed in 636a5b3 / with the release of v0.20.2.

from graylog2-server.