Comments (4)
Yeah. That's a good point.
BufferPool
in Fluency contains some buffer chunks. The sizes of the chunks are multiples of initialBufferSize
. So it's possible some parts of buffer chunks aren't used. And the total size of used buffer chunks is related to whether buffer is full.
Also, the situation of testBufferFullException
is really unusual using StuckSender
. In the test, buffered chunks aren't flushed at all and Fluency keeps to try to expand chunks multiplying by 2.
If we modify the buffer configuration in testBufferFullException
like this, it can use the total allocated buffer up to 87.5%.
PackedForwardBuffer.Config bufferConfig =
new PackedForwardBuffer.Config().setChunkInitialSize(64)
.setMaxBufferSize(2560).setChunkRetentionSize(64);
You can also test it with an actual Fluentd using FluencyTest#testWithRealFluentd
https://github.com/komamitsu/fluency/blob/master/src/test/java/org/komamitsu/fluency/FluencyTest.java#L1005
In this case, the buffer usage was up to 532676608 / 536870912 = 99%.
00:45:55.731 ERROR [pool-3-thread-4] o.k.f.Fluency emit() failed due to buffer full. Flushing buffer. Please try again...
org.komamitsu.fluency.BufferFullException: Buffer is full. config=Config{baseConfig=Config{maxBufferSize=536870912, ackResponseMode=false, fileBackupDir='null', fileBackupPrefix='null', jacksonModules=[]}, chunkInitialSize=1048576, chunkExpandRatio=2.0, chunkRetentionSize=4194304, chunkRetentionTimeMillis=400}, bufferPool=BufferPool{bufferPool={1048576=[], 2097152=[], 8388608=[], 4194304=[]}, allocatedSize=532676608, initialBufferSize=1048576, maxBufferSize=536870912}
at org.komamitsu.fluency.buffer.PackedForwardBuffer.prepareBuffer(PackedForwardBuffer.java:73)
BTW, I found there is a room to improve buffer release a bit in BufferPool
thanks to your report. I'll take care of it later as another pull request. Thanks
from fluency.
I'm assuming this might also be related to why calls to getBufferedDataSize() seems to be larger then actual? I haven't dug into this suspected issue yet it's just been an observation with calling getBufferedDataSize() as part of a background monitoring thread.
from fluency.
@jklap I can't reproduce the issue you wrote above. Can you give me an example code that reproduces that?
from fluency.
I'm closing this issue for now. If you still have any trouble, feel free to reopen it.
from fluency.
Related Issues (20)
- Can we customize the size of data sent? HOT 3
- Exception when ack expected but not received HOT 3
- Some logs are lost after fluentd restart HOT 4
- emit map date issure HOT 2
- ACKs with fluent-bit HOT 8
- Not getting logs on AWS EKS cluster HOT 1
- Cannot use mutual signing TLS HOT 2
- Micrometer metrics HOT 2
- java.lang.NoClassDefFoundError: on Android 6 devices HOT 1
- Introduce strict buffer retention size property HOT 5
- Broken Pipe error after 1 minute of idle HOT 3
- Provide an additional way to configure the RecordFormatter when using the FluencyBuilderForFluentd HOT 4
- Connection reset by peer errors HOT 2
- Need a way to configure fileBackupPrefix of Buffer.Config HOT 3
- Fluency - emit() failed due to buffer full. Flushing buffer. Please try again. HOT 5
- Add support for Unix domain sockets HOT 3
- ackResponseMode true - SocketReadTimeout causes infinite retry HOT 1
- A full buffer should not log any errors HOT 3
- Data loss in the Buffer after throwing an IOException inside `moveRetentionBufferToFlushable()` method HOT 2
- `fluency-fluentd` should expose `fluency-core` via `api` in `build.gradle.kts`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluency.