Fluency fails with SSLException: Broken pipe about fluency HOT 13 CLOSED

@sabarivasan Could you give me the following information?

1. You can connect to the NLB using openssl s_client command?
2. Did you create and specify a JKS file properly?
3. Which Java version did you use?
4. When did the exception happen? Did it always happen when connecting?

from fluency.

1. Yes, I am able to connect to the NLB using the openssl s_client command
2. Yes, we use the same JKS file at other places successfully
3. Java 11

           java -version
          openjdk version "11.0.2" 2019-01-15
          OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
          OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

4. The SSL handshake completes successfully, I believe. It happens when the actual log events are being sent. I turned on -Djavax.net.debug=all to produce detailed logs. I am attaching the relevant portion.

from fluency.

Fluency SSL Bug Low-level traffic log.txt

from fluency.

Thanks! Additional 2 questions:

5. Does this issue happen with other Java versions like 8 as well?

6. Does this issue always happen when it starts sending actual events?

from fluency.

5. Yes, I tried Java 8 and observed the same issue.
6. Yes, it happens when we start sending events, not when the fluency client is created.

from fluency.

Thanks for the quick answer. Let me ask some more...

7. I think NLB provides access logs capturing TLS request information https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html. Can I ask you to get some NLB access logs that capture requests from the Fluency client?

8. I want to know how the Fluentd instance communicated on the TCP connection from Fluency via NLB. Could you capture some TCP packets with, for instance, tcpdump -i eth0 -nX -s512 tcp port 24224?

BTW, Fluency's SSLSender that inherits NetworkSender lazily connects to server at the first send, not at instantiation phase

from fluency.

7. Attached are the NLB log files
   572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2235Z_935609f7.log
   572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200303T1435Z_e188d8e5.log
   572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2235Z_814dfce0.log
   572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2235Z_73c26d4a.log
   572724207364_elasticloadbalancing_us-east-1_net.fluentd-shared-v1-nlb.141e5fc78ae930e9_20200302T2230Z_22292505.log

from fluency.

8. Fluentd runs on a Docker container in ECS Fargate. This means I don't have an EC2 node from which to run the tcpdump command. I am trying to see if there's another way to capture TCP logs.

FYI: I am able to get fluent-bit to send logs to fluentd through this same NLB successfully.

I am also able to send logs from a AWS Lambda written in Javascript using this
library: fluent-logger-node

from fluency.

Thanks again. I looked at the log files, but didn't find anything weird.

I'll create an similar environment to reproduce this issue, but it would be great if there is an endpoint to where I can connect from Fluency on my laptop.

from fluency.

I set up a similar Fluentd environment with NLB + ACM + EC2, but I couldn't reproduced this issue (SSLException: Broken pipe.) However, I ran into an issue that Fluentd behind NLB received invalid packets when both setJvmHeapBufferMode and setSslEnabled are set to true. I suspect this is a bug of Fluency that happens with the combination of SSL and JvmHeapBufferMode.

So, can I ask you to try with setJvmHeapBufferMode(false)? Actually you don't need to call the API since false is the default.

And I noticed your configuration seems a bit too restrict from the perspective of memory resource although it depends your use case. How about trying default values like this to isolate the root cause?

        FluencyBuilderForFluentd builder = new FluencyBuilderForFluentd();
        builder.setSslEnabled(true);
        Fluency fluency = builder.build(host, 24224);

BTW, I'm interested in the throughput, the number of connection reset and so on between your Fluency and Fluentd. Could you share a screenshot of CloudWatch metrics of the NLB on AWS console?

from fluency.

@komamitsu , I just wanted to let you know when I choose defaults for all parameters, it worked!
i then set my original parameters and just turned off JVMHeapBufferMode and things worked.
So it all boils down to ByteBuffer.allocate() vs ByteBuffer.allocatedirect().

Thanks for your support!

I plan to do some load testing on fluency soon. I will send you metrics that you asked for.

from fluency.

Also, is there any documentation on how I can configure the 3 buffer-related params:

bufferChunkInitialSize, bufferChunkRetentionSize and maxBufferSize?

I see that fluency manages a pool of buffers.

If my requirement is to flush the buffer when it reaches 1MB or 30 seconds, whichever comes first, how would I configure this when I create the fluency client?

from fluency.

@sabarivasan Thanks for let me know it. I just fixed the issue at #155 and released it as version 2.4.1.

As for NLB's metrics I wanted to see which side reset the connections to isolate the root cause and I don't think we don't need the metrics now.

If my requirement is to flush the buffer when it reaches 1MB or 30 seconds, whichever comes first

Like this?

        FluencyBuilderForFluentd builder = new FluencyBuilderForFluentd();
        builder.setBufferChunkRetentionSize(1 * 1024 * 1024);
        builder.setBufferChunkRetentionTimeMillis(30 * 1000);
        Fluency fluency = builder.build();

Yeah, I think I should add some more explanations about the buffer configuration. I'll take care of it later.

from fluency.