Comments (13)
@sabarivasan Could you give me the following information?
- You can connect to the NLB using
openssl s_client
command? - Did you create and specify a JKS file properly?
- Which Java version did you use?
- When did the exception happen? Did it always happen when connecting?
from fluency.
- Yes, I am able to connect to the NLB using the
openssl s_client
command - Yes, we use the same JKS file at other places successfully
- Java 11
java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
- The SSL handshake completes successfully, I believe. It happens when the actual log events are being sent. I turned on
-Djavax.net.debug=all
to produce detailed logs. I am attaching the relevant portion.
from fluency.
Fluency SSL Bug Low-level traffic log.txt
from fluency.
Thanks! Additional 2 questions:
-
Does this issue happen with other Java versions like 8 as well?
-
Does this issue always happen when it starts sending actual events?
from fluency.
- Yes, I tried Java 8 and observed the same issue.
- Yes, it happens when we start sending events, not when the fluency client is created.
from fluency.
Thanks for the quick answer. Let me ask some more...
-
I think NLB provides access logs capturing TLS request information https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html. Can I ask you to get some NLB access logs that capture requests from the Fluency client?
-
I want to know how the Fluentd instance communicated on the TCP connection from Fluency via NLB. Could you capture some TCP packets with, for instance,
tcpdump -i eth0 -nX -s512 tcp port 24224
?
BTW, Fluency's SSLSender that inherits NetworkSender lazily connects to server at the first send, not at instantiation phase
from fluency.
- Attached are the NLB log files
572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2235Z_935609f7.log
572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200303T1435Z_e188d8e5.log
572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2235Z_814dfce0.log
572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2235Z_73c26d4a.log
572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2230Z_22292505.log
from fluency.
- Fluentd runs on a Docker container in ECS Fargate. This means I don't have an EC2 node from which to run the tcpdump command. I am trying to see if there's another way to capture TCP logs.
FYI: I am able to get fluent-bit to send logs to fluentd through this same NLB successfully.
I am also able to send logs from a AWS Lambda written in Javascript using this
library: fluent-logger-node
from fluency.
Thanks again. I looked at the log files, but didn't find anything weird.
I'll create an similar environment to reproduce this issue, but it would be great if there is an endpoint to where I can connect from Fluency on my laptop.
from fluency.
I set up a similar Fluentd environment with NLB + ACM + EC2, but I couldn't reproduced this issue (SSLException: Broken pipe
.) However, I ran into an issue that Fluentd behind NLB received invalid packets when both setJvmHeapBufferMode and setSslEnabled are set to true
. I suspect this is a bug of Fluency that happens with the combination of SSL and JvmHeapBufferMode.
So, can I ask you to try with setJvmHeapBufferMode(false)
? Actually you don't need to call the API since false
is the default.
And I noticed your configuration seems a bit too restrict from the perspective of memory resource although it depends your use case. How about trying default values like this to isolate the root cause?
FluencyBuilderForFluentd builder = new FluencyBuilderForFluentd();
builder.setSslEnabled(true);
Fluency fluency = builder.build(host, 24224);
BTW, I'm interested in the throughput, the number of connection reset and so on between your Fluency and Fluentd. Could you share a screenshot of CloudWatch metrics of the NLB on AWS console?
from fluency.
@komamitsu , I just wanted to let you know when I choose defaults for all parameters, it worked!
i then set my original parameters and just turned off JVMHeapBufferMode and things worked.
So it all boils down to ByteBuffer.allocate() vs ByteBuffer.allocatedirect().
Thanks for your support!
I plan to do some load testing on fluency soon. I will send you metrics that you asked for.
from fluency.
Also, is there any documentation on how I can configure the 3 buffer-related params:
bufferChunkInitialSize, bufferChunkRetentionSize and maxBufferSize?
I see that fluency manages a pool of buffers.
If my requirement is to flush the buffer when it reaches 1MB or 30 seconds, whichever comes first, how would I configure this when I create the fluency client?
from fluency.
@sabarivasan Thanks for let me know it. I just fixed the issue at #155 and released it as version 2.4.1.
As for NLB's metrics I wanted to see which side reset the connections to isolate the root cause and I don't think we don't need the metrics now.
If my requirement is to flush the buffer when it reaches 1MB or 30 seconds, whichever comes first
Like this?
FluencyBuilderForFluentd builder = new FluencyBuilderForFluentd();
builder.setBufferChunkRetentionSize(1 * 1024 * 1024);
builder.setBufferChunkRetentionTimeMillis(30 * 1000);
Fluency fluency = builder.build();
Yeah, I think I should add some more explanations about the buffer configuration. I'll take care of it later.
from fluency.
Related Issues (20)
- Can we customize the size of data sent? HOT 3
- Exception when ack expected but not received HOT 3
- Some logs are lost after fluentd restart HOT 4
- emit map date issure HOT 2
- ACKs with fluent-bit HOT 8
- Not getting logs on AWS EKS cluster HOT 1
- Cannot use mutual signing TLS HOT 2
- Micrometer metrics HOT 2
- java.lang.NoClassDefFoundError: on Android 6 devices HOT 1
- Introduce strict buffer retention size property HOT 5
- Broken Pipe error after 1 minute of idle HOT 3
- Provide an additional way to configure the RecordFormatter when using the FluencyBuilderForFluentd HOT 4
- Connection reset by peer errors HOT 2
- Need a way to configure fileBackupPrefix of Buffer.Config HOT 3
- Fluency - emit() failed due to buffer full. Flushing buffer. Please try again. HOT 5
- Add support for Unix domain sockets HOT 3
- ackResponseMode true - SocketReadTimeout causes infinite retry HOT 1
- A full buffer should not log any errors HOT 3
- Data loss in the Buffer after throwing an IOException inside `moveRetentionBufferToFlushable()` method HOT 2
- `fluency-fluentd` should expose `fluency-core` via `api` in `build.gradle.kts`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluency.