incorrect maxBufferSize comparison in BufferPool about fluency HOT 4 CLOSED

Yeah. That's a good point.

BufferPool in Fluency contains some buffer chunks. The sizes of the chunks are multiples of initialBufferSize. So it's possible some parts of buffer chunks aren't used. And the total size of used buffer chunks is related to whether buffer is full.

Also, the situation of testBufferFullException is really unusual using StuckSender. In the test, buffered chunks aren't flushed at all and Fluency keeps to try to expand chunks multiplying by 2.

If we modify the buffer configuration in testBufferFullException like this, it can use the total allocated buffer up to 87.5%.

PackedForwardBuffer.Config bufferConfig =
                               new PackedForwardBuffer.Config().setChunkInitialSize(64)
                                .setMaxBufferSize(2560).setChunkRetentionSize(64);

You can also test it with an actual Fluentd using FluencyTest#testWithRealFluentd https://github.com/komamitsu/fluency/blob/master/src/test/java/org/komamitsu/fluency/FluencyTest.java#L1005

In this case, the buffer usage was up to 532676608 / 536870912 = 99%.

00:45:55.731 ERROR [pool-3-thread-4] o.k.f.Fluency emit() failed due to buffer full. Flushing buffer. Please try again...
org.komamitsu.fluency.BufferFullException: Buffer is full. config=Config{baseConfig=Config{maxBufferSize=536870912, ackResponseMode=false, fileBackupDir='null', fileBackupPrefix='null', jacksonModules=[]}, chunkInitialSize=1048576, chunkExpandRatio=2.0, chunkRetentionSize=4194304, chunkRetentionTimeMillis=400}, bufferPool=BufferPool{bufferPool={1048576=[], 2097152=[], 8388608=[], 4194304=[]}, allocatedSize=532676608, initialBufferSize=1048576, maxBufferSize=536870912}
	at org.komamitsu.fluency.buffer.PackedForwardBuffer.prepareBuffer(PackedForwardBuffer.java:73)

BTW, I found there is a room to improve buffer release a bit in BufferPool thanks to your report. I'll take care of it later as another pull request. Thanks

from fluency.

I'm assuming this might also be related to why calls to getBufferedDataSize() seems to be larger then actual? I haven't dug into this suspected issue yet it's just been an observation with calling getBufferedDataSize() as part of a background monitoring thread.

from fluency.

@jklap I can't reproduce the issue you wrote above. Can you give me an example code that reproduces that?

from fluency.

I'm closing this issue for now. If you still have any trouble, feel free to reopen it.

from fluency.